This is kind of random but- I wonder, if you had a sufficiently complex lens, or series of lenses, perhaps with specific areas darkened, could you make a lens that shone light through if presented with, say, a cat, but not with anything else? Bending light and darkening it selectively could probably reproduce a layer of a neural net. That would be cool. I suppose, you would need some substance that responded to light in a nonlinear way.
But yes, you are restricted to linear things and you can't make a good photonic cat detector out of that easily. So all the photonic neural networks you may have heard of like https://arxiv.org/abs/2106.11747 wind up sticking some mechanical or electrical nonlinearity somewhere.
You can simulate materials, apply the wave equation, and get "layers" that compute outputs from given inputs, each modeled as points in space. It may be possible to manufacture such layers with metamaterials or something like that.
A research out of ETH Zurich, based on which the company Rayform https://rayform.ch/ was founded does exactly this! I was so excited when I saw the paper for the first time a couple of years ago.
