I started thinking about how you might automate game development, and I think it's a pretty good thought excercise on the topic.
I feel like the end result is just what we already have, game engines with visual programming and drag and drop editors. You don't add much value by automating the programming since you still have to define the input and outcome.
The real win was building the complex logic and state editor into a good UX.
Predicting the next frame from previous frames and learnt sequences is a neat trick, but I think automated game development is already very possible with simple techniques like genetic algorithms, or even a PRNG. I mean Rogue and it's descendants are very much automated game development, but GPT-3 could be useful for something like dynamic quest generation, world-building, narrative, adaptive NPCs (including dialogue) etc.
I've seen it before, and I've a basic understanding of GANs, I just don't see it being overly useful. This technique can make a really blurry simulacrum of an actual game, and that's really cool, but I'm not sure how it could be used to make a something both truly novel and coherent. There's plenty of low hanging fruit for AI within an engine, whereas using AI to be the entire engine is somewhat infeasible
You are right about the technique not being very useful as of today. My fascination stems from my assumption that more resources poured into this approach would yield vastly better results.
Even just watching the video, I came up with several possible improvements to try out. Eg adversarial training, that would really hone in on the situations and aspects where the model is weak so far, like edge conditions; instead of just using normal gameplay as input.
It's definitely an interesting area of research, but for that example you still had to make the whole game in the first place in order to have something to train the model on. Say you have a novel game idea, how could you use that approach to make it a reality? I'm not sure you could, but like you mention it's a really early example and who knows where it ends up.
The other part about that GAN Theft Auto example is that it doesn't actually know what's going on, like there's no game state. All it knows is that "When I have a frame that looks like this, and they press that button, I think the next frame would usually look like this". So it's got no internal game logic, it's just really good at painting what games look like.
Even going about this very naively, you could at least use it to train a model against a supercomputer running the game, and then run the inference on much more modest end-user machines.
But you can be much more ambitious: have you seen eg style transfer? So you could probably do a bit of ML black magic to train your model on GTA, and then point it at the Google Earth data to get a GTA-like set in real-life London.
Or you could use something like style transfer to go for a cartoony look, or add ray-tracing like effects, even if you didn't have these effects in your original engine.
Or you can use a pre-trained model (eg on GTA), and then spend a relatively modest amount of extra training to get a different kind of game, eg one that has magic or so.
About the latter part: I do think their model is already running with some state. But even if it ain't, that's a relatively small thing to add with already known standard techniques (or you can come up with new techniques.)
And this is where I think of the game-side of games programming to be more an art form (I say this as an ex-professional game developer). GPT-3 could be used for narrative generation, environment generation, and adaptive AI, and these are all exciting areas for research and experimentation. But as you say, underneath is a solid engine, and one day an AI could feasibly build an engine from scratch, but I think that day is decades away. The AI that we have today are just toys.
I feel like the end result is just what we already have, game engines with visual programming and drag and drop editors. You don't add much value by automating the programming since you still have to define the input and outcome.
The real win was building the complex logic and state editor into a good UX.