Covariance and contravariance also have meanings in mathematics, for which this seems like a specialization though I don't have a concrete connection.
In math you often have a class of objects and functions on those objects (say, groups with group homomorphisms, or simpler, types in a programming language with functions that take one type to another). Then in math one often learns about an object by associating another object with it. For example, a topological space can be associated with a specific kind of group that tells you about the space's structure.
Then you can ask how this "association" behaves with the functions on the object. I'll use notation now, and say that for an object X the "associated" object is G(X). Often G carries over to functions, so that if f is a function X -> Y then G(f) is a function G(X) to G(Y). Such an "association" is called covariant because it preserves the ordering of domain -> codomain. The "association" is instead called contravariant if it reverses the order. E.g., f: X -> Y becomes G(f): G(Y) -> G(X). Of course, I'm using "association" but the real word for G is functor.
A generic type with a covariant parameter can be seen as a covariant functor if you pick the right category.
Specifically you need the category where the objects are types and the morphisms indicate inheritance (i.e. A -> B means that A is a subtype of B).
Then a functor F is covariant if A -> B implies F(A) -> F(B). Which is precisely what it means to be a covariant generic type.
Note: you shouldn't confuse this category with the category where the objects are types and the morphisms are functions that map values of one type to values of another type.
If I'm not mistaken, considering the category of subtyping relationship you get exactly that: with the covariant functor you get (in Scala notation) if B <: A then F[B] <: F[A] and with the contravariant functor if B <: A then F[B] >: F[A]
Interestingly, in physics covariance and contravariance of vectors are basically on equal footing. This is pretty different the OP usage for which covariance and contravariance denote either toward or away from the root of a tree (the object hierarchy).
Like you, I'd be interested to know whether there's a deeper connection in the terminology.
So I'm just a computer scientist, not a mathematician or phycist, but I don't think there is a direct connection. The words co/contravariance is used for two different things in mathematics, vectors in physics/differential geometry, and functors in category theory. (https://en.wikipedia.org/wiki/Covariance_and_contravariance).
The computer science meaning is inspired by category theory (and can be seen as a simple instance of it). But the vector/functor meanings seem to be unrelated.
There is a bit of a connection, because the operation ٭ which takes a vector space V to the set of 1-forms V٭ is a contravariant functor. (I.e., linear maps V→U give maps of 1-forms U٭→V٭). But in physics, we say that the members of V are contravariant and the members of V٭ are covariant, i.e. the physics notion is about individual spaces, not about mappings between spaces.
My understanding, though I've never personally made it explicit, is that you view "contra/covariance" as arising not as a property of vector spaces exactly (concretely) but instead from tensors-as-multilinear-maps and you talk about them as functors between transformation groups on vector spaces which either preserve (co-) or reverse (contra-) transformations.
The inner product lets you view vector spaces as spaces of transformations already so the same sort of idea comes out now viewing vector spaces as maps in their own right.
In math you often have a class of objects and functions on those objects (say, groups with group homomorphisms, or simpler, types in a programming language with functions that take one type to another). Then in math one often learns about an object by associating another object with it. For example, a topological space can be associated with a specific kind of group that tells you about the space's structure.
Then you can ask how this "association" behaves with the functions on the object. I'll use notation now, and say that for an object X the "associated" object is G(X). Often G carries over to functions, so that if f is a function X -> Y then G(f) is a function G(X) to G(Y). Such an "association" is called covariant because it preserves the ordering of domain -> codomain. The "association" is instead called contravariant if it reverses the order. E.g., f: X -> Y becomes G(f): G(Y) -> G(X). Of course, I'm using "association" but the real word for G is functor.