I guess it was used as to mean "without value", which is not exactly what it means.
The prefix "in" of "invaluable" does mean negation but it means negation in the way that the value of the object is such that it doesn't exist. It's so valuable that you cannot even quantify it, it transcends the notion of value.
interestingly, you can track this issue back to latin, which has a preposition "in" (meaning in, on, to), which is often attached to verbs, and a prefix "in-" which negates the word it's attached to.
invaluable (adjective): extremely useful