You can render a video file as a volume. I've looked at using that to make a video compression algorithm that operated on the volume rather than on the 2D frame stream. My hunch was that shapes in the 3D volume changed more predictably than surfaces from frame to frame because the frames describe the movement of objects in space. But they're projected onto a two dimensional surface. So you get these interesting 3D shapes that have fairly predictable qualities across larger spans of time than your average 2D encoder sees while encoding a video. But I never could get it to work more efficiently than existing algorithms.
I suspect you'd need your compression to "understand" the object relationships and camera movement to do better than frame sequences, and it'd probably still be incredibly hard because you then add a lot of extra information first in the hope they let you discard more pixel data...
But the more you understand the scene, the more you can potentially outright reconstruct, and in some contexts more loss would be entirely fine if the artifacts are plausible.
That's exactly where I ended with this: I was decomposing the scene and realized that if you had the ability to do that reliably enough you'd be recreating a model rather than an image and then re-rendering that model. But at that point I don't think you are looking at a compression algorithm any more other than in the very broadest sense of the word. Boundaries between objects would start to look fuzzy otherwise. As in: you'd no longer know exactly where the table ended and the hand started unless you modeled it precisely enough and at that point you have an object model. So you might as well use it to render the whole scene.
Note that I did this in '98 or so, when there was less of a computational budget, maybe what I couldn't hack back then is feasible today.
Still, it was a fun project.