Traditionally, if you have a real scene that you want to be able to reproduce as 3D graphics on the computer, you either:
1. Have an artist model it by hand. This is obviously expensive. And, will be stylized by the artist, have a quality levels based on artist skill, accidental inaccuracies, etc...
2. Use photogrammetry to convert a collection of photos to 3D meshes and textures. Still a fair chunk of work. Highly accurate. But, quality varies wildly. Meshes and textures tend to be heavyweight yet low-detail. Reflections and shininess in general doesn't work. Glass, mirrors and translucent objects don't work. Only solid, hard surfaces work. Nothing fuzzy.
Splatting is an alternative to photogrammetry that also takes photos as input and produces visually similar, often superior results. Shiny/reflective/fuzzy stuff all works. I've even seen an example with a large lens.
However the representation is different. Instead of a mesh and textures, the scene is represented as fuzzy blobs that may have view-angle-dependent color and transparency. This is actually an old idea, but it was difficult to render quickly until recently.
The big innovation though is to take advantage of the mathematical properties of "fuzzy blobs" defined by equations that are differentiable, such as 3D gaussians. That makes them suitable to be manipulated by many of the same techniques used under the hood in training deep learning AIs. Mainly, back-propagation.
So, the idea of rendering scenes with various kinds of splats has been around for 20+ years. What's new is using back-propagation to fit splats to a collection of photos in order to model a scene automatically. Before recently, splats were largely modeled by artists or by brute force algorithms.
Because this idea fits so well into the current AI research hot topic, a lot of AI researchers are having tons of fun expanding on the idea. New enhancements to the technique are being published daily.
Thanks for the explanation. Reading the paper, it seems it still takes 2-4 hours to train (one scene?). I imagine that's still faster than any manual method.
1. Have an artist model it by hand. This is obviously expensive. And, will be stylized by the artist, have a quality levels based on artist skill, accidental inaccuracies, etc...
2. Use photogrammetry to convert a collection of photos to 3D meshes and textures. Still a fair chunk of work. Highly accurate. But, quality varies wildly. Meshes and textures tend to be heavyweight yet low-detail. Reflections and shininess in general doesn't work. Glass, mirrors and translucent objects don't work. Only solid, hard surfaces work. Nothing fuzzy.
Splatting is an alternative to photogrammetry that also takes photos as input and produces visually similar, often superior results. Shiny/reflective/fuzzy stuff all works. I've even seen an example with a large lens.
However the representation is different. Instead of a mesh and textures, the scene is represented as fuzzy blobs that may have view-angle-dependent color and transparency. This is actually an old idea, but it was difficult to render quickly until recently.
The big innovation though is to take advantage of the mathematical properties of "fuzzy blobs" defined by equations that are differentiable, such as 3D gaussians. That makes them suitable to be manipulated by many of the same techniques used under the hood in training deep learning AIs. Mainly, back-propagation.
So, the idea of rendering scenes with various kinds of splats has been around for 20+ years. What's new is using back-propagation to fit splats to a collection of photos in order to model a scene automatically. Before recently, splats were largely modeled by artists or by brute force algorithms.
Because this idea fits so well into the current AI research hot topic, a lot of AI researchers are having tons of fun expanding on the idea. New enhancements to the technique are being published daily.
https://radiancefields.com/