Just to back this point up, the single biggest challenge to headset makers is power management.
Every time you move a byte one stage away from the sensor you (general) incur an order of magnitude of power cost. There are many many caveats, and its not a hard and fast rule, but its a useful illustration. Don't get up about precise numbers, just understand that moving data between devices, even inside a processor has a power cost.
e.g. reading out a pixel from a camera costs 1, moving it to a UART/interface costs 10, reading it into a register costs 100, l1 1000 etc etc etc.
The closer you can do the processing to the sensor the greater the (potential) power saving, and in theory the lower the latency.
For example the eye cameras you only really want the direction the eyes are looking. So you don't want to ship the entire image at 120fps to a processor, do some maths to then get that vector. Ideally you want the sensor to do it for you and just ship four floats every 1/120th of a second.
I'm also not sure where the main processor is for the goggles, so data compression and coordination of the sensors also becomes critical. The images sensors almost certainly don't run at a high enough framerate to be "magical" so you need IMU to generate a fudge factor. They need to run at a known sample rate, one that doesn't deviate. Using the CPU to do that is bad, because that'll require a lot of interrupts, leaving little room for other stuff that's important (like other sensors)
now, I've hinted that the "SPU" does processing, which is almost certainly does. This means that it might actually look like a GPU after all, as its doing a lot of calculations that are very "graphics-y" All that computer vision: SLAM for head orientation, HAnd tracking for, well, hands, Image stitching/warping and correction to make the pass through work, plus the audio shit, it all very computer vision. So its highly likely that it looks a bit like a GPU, because its doing similar tasks
You seem knowledgeable about this stuff — do you work in this space? I’m trying to do some feature extraction from multiple global shutters right now on limited hardware and its a learning experience!
Every time you move a byte one stage away from the sensor you (general) incur an order of magnitude of power cost. There are many many caveats, and its not a hard and fast rule, but its a useful illustration. Don't get up about precise numbers, just understand that moving data between devices, even inside a processor has a power cost.
e.g. reading out a pixel from a camera costs 1, moving it to a UART/interface costs 10, reading it into a register costs 100, l1 1000 etc etc etc.
The closer you can do the processing to the sensor the greater the (potential) power saving, and in theory the lower the latency.
For example the eye cameras you only really want the direction the eyes are looking. So you don't want to ship the entire image at 120fps to a processor, do some maths to then get that vector. Ideally you want the sensor to do it for you and just ship four floats every 1/120th of a second.
I'm also not sure where the main processor is for the goggles, so data compression and coordination of the sensors also becomes critical. The images sensors almost certainly don't run at a high enough framerate to be "magical" so you need IMU to generate a fudge factor. They need to run at a known sample rate, one that doesn't deviate. Using the CPU to do that is bad, because that'll require a lot of interrupts, leaving little room for other stuff that's important (like other sensors)
now, I've hinted that the "SPU" does processing, which is almost certainly does. This means that it might actually look like a GPU after all, as its doing a lot of calculations that are very "graphics-y" All that computer vision: SLAM for head orientation, HAnd tracking for, well, hands, Image stitching/warping and correction to make the pass through work, plus the audio shit, it all very computer vision. So its highly likely that it looks a bit like a GPU, because its doing similar tasks