I'm not sure where they're getting that "lenses in pathtracers are ignored". They most certainly are not. How would physically based path tracing even work if they were?
Most commercial renderers that I know of just render via simulating a pin-hole camera. Any Camera lens effects, lens flare, bokeh is just simulated off of the very simple pin-hole model. The method in Lentil appears to be actually trying to simulate the ray bounces internal to the camera lenses via the geometry from patent information. I am not an expert on optics, but from what I know about modern lens design are these elements are coated to change reflection and designed to prevent optical problems that don't exist in most computer renders (Chromatic Aberration). My initial take is this simulation incorrectly solves a problem that most users rendering things have no interest in even dealing with. Computer rendering has moved into this simulating the physical world but without asking why, in many rendering cases you do not want a camera that behaves like the physical world, you want it to behave better and with less control.
The main use case that popped in my head was compositing of rendered effects with live footage from a physical camera. In this case, might it be beneficial for your render to exactly match the real world lens, defects and all?
It is typically done in compositing since it is impractical to render with depth of field which gives you a lot more noise and takes away your ability to change it without a re-render.
I think most commercial renderers offer a thin lens model in addition to pin-hole to get DoF and custom bokeh, but using a real multi-lens setup gets you some more realistic effects.
Traditionally most cg rendering systems use an idealized lens to simplify the math. Matrix projection transformations simulate a pinhole lens.
This does mean that all images are perfectly sharp however, so for depth-of-field, bokeh, and other realistic lens blurring effects the position of the idealized lens (i.e. virtual pinhole) is varied over samples creating a sort of fuzzy pinhole.
The name makes sense if you're aware of the etymology of the word "lentil" [1]:
From Middle English lentile, from Old French lentille from Latin lenticula, diminutive of lēns, from a Proto-Indo-European root shared by German Linse and Ancient Greek λάθυρος (láthuros). Doublet of lenticula.
In Swedish, for example, the word for the food ("lins") is exactly the same as the word for the optical component. I just thought that was interesting, since naming is hard and perhaps the connection was not obvious to everybody.
Lentil is an advanced camera toolkit for the Arnold renderer. It is a set of shaders that extends creative control and physical correctness, while significantly reducing the time (~30x) needed to render bokeh through bidirectional sampling of camera rays.
Efficiently simulating realistic lens systems has been a thing for a while; the original paper by Kolb, Mitchell, and Hanrahan is here: https://www.cs.utexas.edu/~fussell/courses/cs395t/lens.pdf. In the paper they talk about how to apply importance sampling, which is the key to efficiency in bidirectional path tracing. What does this package offer beyond that?
They claim to be 30x faster than the Arnold CPU renderer, but of course most people use the GPU renderer, which this tool doesn't support and which is potentially 100x faster.
So they are comparing their main speed claim against an outdated and known slow implementation.
Also, vray (another renderer) ships with a variety of lens shaders and if I recall correctly already decouples main rays and camera focus rays.
What I agree on is that having more realistic lens shaders is great and that they should be fast to calculate. But since doing bokeh in post is good enough for Hollywood movies, I'm not sure if their solution offers a large enough increase in image quality to justify the surely large increase in rendering costs.
I was imagining some kind of really wacky setup where you fire light from a simulation through a real lens to get realistic lens effects. But what kind of display would you need?
You can. It's just very costly. Rays typically represent just a single sample, so the more probabilistic the projected image becomes (read "unfocused"), the more samples are required to capture it smoothly. This is compounded by the number of lens elements, the material's index of refraction and other complexities. A single element lens adds a minimum of two additional ray/surface evaluations for every sample of which their are many per pixel.
Also, as a separate matter many implementations optimize around the assumption that the projection from three dimensions to two is done in a uniform, undistorted space. For example the kind of projection that can be fully described by a 4x4 matrix. Changing that assumption loses those optimizations, and generally leads to substance abuse and mayhem.
Not an expert, I would guess at least these two things:
1. There are actually around 20 lenses in a camera, not one.
2. A lens converts a ray into a cone, right? And then you need to pass that cone through the other 20 lenses which further modify it's shape, differently at different wavelengths.
I would guess this is extremely computationally expensive done naively.
Cone tracing is a thing that exists, but I think most path tracers just just regular rays. (There's also ray differentials, which if I understand correctly, are a way of tracking how a cone of light would spread out after bouncing off of curved surfaces, if the ray were a cone.)
As to why lenses would cause a problem specifically, I don't know the path tracing algorithm well enough to say for sure offhand, but it may have something to do with introducing random sampling before you even hit the first scene object. Usually in ray tracing the first hit is kind of a freebie; you can calculate direct illumination exactly, and it's only indirect illumination that's approximated. (In path tracing, it's approximated by doing a lot of random sampling.) So, not having to approximate the first-hit direct illumination reduces the noise quite a bit right off the bat.
Most modern render engines render rays from the camera into the scene. This speeds up rendering a lot but also means rendering caustics will need a huge amount of samples because the probability of seeing light from a camera through glass is very low. Bi-directional path tracing fixes some of these problems.
So it can be done but always at a cost (rendering times).
In Blender I have done this a couple of times and even used different IORs for the RGB colors to get real world results.
Is there any kind of standard file format describing lens internals, or database of different lens designs or actual commercially available lenses? I'm interested in this kind of simulation for some machine vision applications that may need to cope with not all of the image being in perfect focus at once.
Path tracing is currently state-of-the-art for making photorealistic CG images because it can generate real reflections, refractions &c. by simulating something very similar to how the physical world works.
A photo (or movie) is, however, taken through a lens that distorts the image (sometimes intentionally; what is and isn't in focus in a composition is important for drawing the viewers attention).
A path tracer can already simulate this by just putting a bunch of pieces of glass shaped like the real lenses of a camera, but this is extremely computationally intensive.
This particular implementation uses a few (already known in the literature) techniques for making it faster: bidirectional tracing, selective oversampling, and thin-lens approximations.
This is an implementation of these techniques for Arnold, which is a renderer implementation used for many feature films[1].
Probably not much - the reality is that rendering depth of field in camera in production CG is a very tough sell. Even if there is no speed penalty, renders are usually so expensive that it isn't even a question of doing it in real time in compositing.
That video has nothing to do with this. This plug-in is just a sampling strategy to make rendered depth of field less noisy.
The video you linked just talks about a shot that was purposely made to be an homage to a classic cinematic technique even though it isn't necessary. Depth of field in production computer graphics is almost exclusively done with depth maps in compositing. Because there are many different layers that won't give seam artifacts around the edges as well as floating point outputs that don't clip, depth maps are even easier to make work.
Arnold has both CPU and GPU rendering support. It is used notably by Sony Pictures Imageworks and created my Marcos Fajardo now working at Solid Angle.
e.g. https://blog.demofox.org/2018/07/04/pathtraced-depth-of-fiel...