No. Fair point indeed - the tires are not the only difference. I suspect they are one of the biggest though (based on subjective feeling - for what it’s worth).
It's almost physically impossible for the tires to be a bigger difference than geometry (and thus your position), specially at faster speeds as you mentioned, because aerodynamics quickly dominate the overall resistance you're fighting to keep your speed.
