If we take USB4 80 Gbps as a guideline: That's 80 mbit per millisecond. Apple says their goggles have 23 mpixels. So if we multiply by 24 bits for color, transmission has a lower bound of 7 ms per frame.
So I agree now that the delay is in milliseconds which can push perception into nausea. I hadn't considered the pixel density. Of course they could use something faster than USB4, but it wouldn't be easy to go below 1 ms I guess.