It depends on what parameters you give the system control over. It doesn't necessarily have to generate the code itself, it can simply control one of the many other things. For example, if you only give control over some dimensional parameters like the command line flags, or something like warp/block/thread scheduling parameters (e.g. GPU programming), then you could maybe ask questions like "Given this block of code, running on this hardware, what are the best compilation flags and build parameters."
There was an interesting paper a while back before the LLM craze that basically did this for OpenMP programs. You'd feed it the source code to some loop body or whatever, and it would try to pick the right OMP block/loop scheduling parameters from there.
Naturally, if you're training it on open code, you will have to censor the AI so it doesn't use flags like -ffast-math too much. :)
