When I was working on a game engine, I came across a few papers trying to do the same thing. I think it's more than worthy of academic study. It's no different than any other compiler research.
The problem isn't just compiling C++ to glsl. The problem is taking c++ and allowing you to write your shader as part of your program. Right now, you essentially have 2 options: you can write one shader that does everything and use the actual data passed to it to conditionally run what you want, or you build a shader composition system that takes small chunks of code and pieces them together based on what data you want to pass to it. Thats overly simplified, but a kind of high level view. There are a few talks from bungie detailing the work they put into their shader system so that technical artists could have a smooth workflow.
Ultimately, your engine can't handle custom shaders without the engine user also writing code for the cpu side, unless you have a AAA sized engine team that develops a compiler to achieve that.
Hi, lead author here. You're definitely getting at the core of the issue. An effective shader programming models needs to support modularity and composition of shader code, while also supporting GPU code specialization and interop between host and GPU code. Large game engines provide layered tools that aim to provide these features, and they are very useful especially for users who aren't experts in graphics programming (e.g., artists and technical artists). I'll use this opportunity to shamelessly plug our prior work, which explores these aspects of shader systems in more detail: https://dl.acm.org/doi/10.1145/3355089.3356554 (preprint here: https://escholarship.org/uc/item/2f8448n2 )
Even with these layered tools, the host-GPU interop is still something that shader programming systems have to contend with directly, in contrast to "unified" systems like CUDA and C++ AMP in the GPU compute world. As you point out, translating C++ to GPU-compatible code is just one step in the process. Our work looks at the next step: once we can write both host and GPU shader code in C++, what's missing? What we discovered is that by merging the host and GPU halves together in C++, we inadvertently broke the critical shader specialization optimization. So our work presents a method to support shader specialization in a first class way, while also meeting our other design goals.
The problem isn't just compiling C++ to glsl. The problem is taking c++ and allowing you to write your shader as part of your program. Right now, you essentially have 2 options: you can write one shader that does everything and use the actual data passed to it to conditionally run what you want, or you build a shader composition system that takes small chunks of code and pieces them together based on what data you want to pass to it. Thats overly simplified, but a kind of high level view. There are a few talks from bungie detailing the work they put into their shader system so that technical artists could have a smooth workflow.
Ultimately, your engine can't handle custom shaders without the engine user also writing code for the cpu side, unless you have a AAA sized engine team that develops a compiler to achieve that.