Somewhat aside, but the brain having to "flip" visual information because it's "upside down" seems suspect to me. Turn it sideways while maintaining all the connections it has to the rest of the body, and what changes? Is it getting visual information sideways that it has to rotate now? Probably not.
Moreover the idea that the collection of neurons that your retina connects to has any concept of "orientation" is nonsense to begin with IMO. It's not that "there's an upside-down image that your brain has to fix", it's just that your brain interprets signals from your retina as a picture in your mind, full stop.
Rods/cones in the top of your retina connect to your brain through neurons, so do the ones at the bottom. But to say that "this 'top' retinal cone should really connect to a 'top' neuron in your brain", doesn't even make sense to me. Since when do the locations of the neurons interpreting the input even matter?
It would be the same with hearing too... you have a left and right ear, but if for some reason those were swapped and your left fed things to the right half of your brain and vice-versa, your brain wouldn't be "flipping it back", because how could the absolute location of the neurons interpreting the sounds even matter?
This is the right way to look at it. In fact, your brain is plastic enough that if you wear glasses that flip your vision upside down for several days it will eventually relearn the mapping of retinal cells to neurons so that you see things normally while wearing them. This was studied in the 1890s by a guy called George Stratton.
"Since when do the locations of the neurons interpreting the input even matter?"
Incidentally, these neurons theoretically could go anywhere (as long as they're connected correctly), but in practice they end up arranged retinotopically (https://en.wikipedia.org/wiki/Retinotopy).
People don't really understand what the terms 'up' or 'down' mean, in general. Down is the direction of the sum of the gravitation forces. That's why things fall down.
The brain doesn't care about up or down location of nerves, because it is not affected much by gravity, it is held in place. If a nerve is above or below another, the brain doesn't care.
On a side note, you should also realize that the direction 'down' is dependent on your location, unless you are on the discworld.
