Could someone explain what splatting is suitable for? I see lots of recreations of photographs. Is this some a kind of compression technique, or is there some other usage?
Traditionally, if you have a real scene that you want to be able to reproduce as 3D graphics on the computer, you either:
1. Have an artist model it by hand. This is obviously expensive. And, will be stylized by the artist, have a quality levels based on artist skill, accidental inaccuracies, etc...
2. Use photogrammetry to convert a collection of photos to 3D meshes and textures. Still a fair chunk of work. Highly accurate. But, quality varies wildly. Meshes and textures tend to be heavyweight yet low-detail. Reflections and shininess in general doesn't work. Glass, mirrors and translucent objects don't work. Only solid, hard surfaces work. Nothing fuzzy.
Splatting is an alternative to photogrammetry that also takes photos as input and produces visually similar, often superior results. Shiny/reflective/fuzzy stuff all works. I've even seen an example with a large lens.
However the representation is different. Instead of a mesh and textures, the scene is represented as fuzzy blobs that may have view-angle-dependent color and transparency. This is actually an old idea, but it was difficult to render quickly until recently.
The big innovation though is to take advantage of the mathematical properties of "fuzzy blobs" defined by equations that are differentiable, such as 3D gaussians. That makes them suitable to be manipulated by many of the same techniques used under the hood in training deep learning AIs. Mainly, back-propagation.
So, the idea of rendering scenes with various kinds of splats has been around for 20+ years. What's new is using back-propagation to fit splats to a collection of photos in order to model a scene automatically. Before recently, splats were largely modeled by artists or by brute force algorithms.
Because this idea fits so well into the current AI research hot topic, a lot of AI researchers are having tons of fun expanding on the idea. New enhancements to the technique are being published daily.
Thanks for the explanation. Reading the paper, it seems it still takes 2-4 hours to train (one scene?). I imagine that's still faster than any manual method.
One big reason why people are excited is that it's finally a practical way to synthesize and render complete photorealistic 3D scenes without any of the traditional structural elements like triangle meshes and texture maps.
Think of the difference between vector graphics (like SVG) and bitmap graphics (like JPEG or PNG). While vectors are very useful for many things, it would be quite limiting if they were the only form of 2D computer graphics, and digital photos and videos simply didn't exist. That's where we have been in 3D until now.
Probably the biggest real life application is 3D walkthroughs of houses for sale & rental. This already exists, but the quality isn't as good as shown here.
Other examples are things like walking through archeological sites, 3D virtual backgrounds (e.g. for newsrooms), maybe crime scene reconstruction?
It's basically perfect 3D capture, except the big limitations are that you can't change the geometry or lighting. The inability to relight it is probably the most severe restriction.
It is likely the future of compositing and post-processing. Instead of blue screens, you can capture actors and real life items and blend them seamlessly with CGI stages and props (relighting is likely coming soon too). Additionally, you can reframe, add or remove elements from the scene in post-production essentially for free.
Novel view synthesis, so based on some images of a scene, rendering views from positions and angles that were not originally recorded. 3D Gaussian Splatting, besides beating the state-of-the-art in terms of visual quality at its time of release, also has some nice physical properties like having the splats associated with actual points in 3D (obtained through feature extraction).
It's useful for help in planning just about any work that is done outdoors, since it gives you a digital version of reality, just like photogrammetry. Think agriculture, construction, architecture, etc. Many environments can not be photographed in their entirety because of the laws of physics. Here, a digital recreation of reality is the only way to get an accurate picture. For example underwater environments and caves.
3d navigation, lidar-like maps of a space just based on pics, to allow a drone to beable to have a lidar-like spatial awareness from 2d imagery? (aside from the cool photograpgy bits it offers)
