"Don’t ask me why, but copying pixel data out of a video tag is expensive"
I'd bet good money that is because it has to switch to a software RGB system-memory-backed indirect rendering path, bypassing all the acceleration provided by modern GPUs, whether that is on-GPU H.264 decoding, or just YCrCb colorspace conversion, or any of the other tricks that standard 2d video players pull off. Or it might even try to do the full HW path and then read back RGB "in the opposite direction" from the final framebuffer which would be even slower.
I'd bet good money that is because it has to switch to a software RGB system-memory-backed indirect rendering path, bypassing all the acceleration provided by modern GPUs, whether that is on-GPU H.264 decoding, or just YCrCb colorspace conversion, or any of the other tricks that standard 2d video players pull off. Or it might even try to do the full HW path and then read back RGB "in the opposite direction" from the final framebuffer which would be even slower.