Certainly there is some inductive bias. Neuroanatomy is broadly similar human to human which constraints connectivity. There is an energetic penalty for excess wiring - evolution drives neuron placement to minimize wiring length (eg topographic layout of layers in cortex), which further constraints connectivity. In general, there is a huge body of evidence supporting the hypothesis that organisms are highly tuned to their environments.
My point is that in contrast to fruit flies, humans (well, mammals) raised in environments with non-natural scene statistics show significant differences in coding in the visual cortex. Here are some random references I dredged up:
It's not hard to see why this might be selected for in mammals. The historic range of D. melanogaster is just sub-saharan Africa, whereas the historic range of humans is huge, and we are primarily visual organisms. So it would be advantageous to have different tuning if you lived in the desert vs the forest. Of course you have to look at the range and divergence times of the full phylogeny but I am getting distracted from my actual research. It's a very interesting line of thinking though. I was quite surprised to learn things were hardcoded in flies when I joined a fly vision lab.
I think there is a big gap between the two claims -
1. Human visual cortex is very plastic
2. There is no inductive bias in the visual part of the brain that has been "trained" by millions of years of evolution.
I trust you are correct on the former, but the latter does not seem to follow from the former.