Ok so I've tried doing some version of this that is a bit more advanced [0] but I gave up because I'm not a ML expert. Have you thought about creating/projecting video versions of lightfields? Like Google's Deepview [1]. I'd love for DeepView Video kind of tech to be comoditized.
I worked on a lightfield(ish) camera at Facebook, but Lifecast is more focussed on what is practical with current camera hardware. We prefer to make the best possible 6DOF using existing VR180 cameras which people already have. A second challenge is to render the results on a Quest 2 (the most popular VR headset today), with its limited GPU power. Our format is optimized for the rendering capabilities of Quest 2, which means we have to make some tradeoffs on visual quality. I don't think Quest 2 has enough power to render multi-layer images (MPIs, the format in DeepView). This is the difference between making a product and doing academic research. I'm looking forward to Quest 3 or whatever comes next; I hope it has enough power to do stuff like MPI.
[0] https://roblesnotes.com/blog/lightfields-deepview/
[1] https://augmentedperception.github.io/deepviewvideo/