That's exactly where I ended with this: I was decomposing the scene and realized...

That's exactly where I ended with this: I was decomposing the scene and realized that if you had the ability to do that reliably enough you'd be recreating a model rather than an image and then re-rendering that model. But at that point I don't think you are looking at a compression algorithm any more other than in the very broadest sense of the word. Boundaries between objects would start to look fuzzy otherwise. As in: you'd no longer know exactly where the table ended and the hand started unless you modeled it precisely enough and at that point you have an object model. So you might as well use it to render the whole scene.

Note that I did this in '98 or so, when there was less of a computational budget, maybe what I couldn't hack back then is feasible today.