On: Trilinear point splatting for real-time radiance f...

[heliophobicdude]

If anyone is interested in this field, we need your help with real-time reconstruction.

Currently a huge challenge is in real-time reconstruction. Approaches involve estimating point clouds from images then optimizing splats on those. Other approaches are using SLAM like LiDAR to have distance of points to the camera but then still optimizing on that.

Optimization is producing good results but takes iterations that are not suitable for real-time.

Pixel-wise splat estimation with iPhone LiDAR could produce good results but need help and expertise

[heliophobicdude → heliophobicdude]

If anyone is up to the challenge, Apple provides an excellent sample project available for download with low level access and operations on the LiDAR points here and shading all for an iPhone app:

https://developer.apple.com/documentation/avfoundation/addit...

[dwallin → heliophobicdude]

Is this paper achieving what you mean: https://arxiv.org/pdf/2311.16728.pdf

Caveat, this field is far outside my wheelhouse. So if not, I'd love to understand more about the nuances here.

[markisus → dwallin]

I don’t think this is realtime reconstruction, meaning that changes in the scene will not be represented in the reconstructed map immediately.

[smusamashah → markisus]

The Splatter Image paper recently featured on HN seems to do that though https://news.ycombinator.com/item?id=38720226

[smusamashah → heliophobicdude]

How about Splhow about https://szymanowiczs.github.io/splatter-image it works for single image to 3d but claims that adding more images improves the quality

[londons_explore → heliophobicdude]

You say you need help... But doesn't this paper solve the problem? Ie. 60fps and quality looks pretty much perfect to me. What's next to do?

[brookman64k → londons_explore]

The paper says that training / reconstruction takes 4 hours. While inference runs at 60fps reconstruction is far from real-time.

[countWSS → heliophobicdude]

Perhaps synthetic generation could help? something like InseRF or Instruct-NeRF2NeRF could be used to generate components and variations.