I imagine this might be interesting for domain-specific GPT models. Say training it on a mountain of technical documentation, or on every fanfiction published on the internet, or a sentiment analysis dataset. Of course fine-tuning GPT3 would give better results, but nanoGPT might allow you to make a much smaller model that's still good enough, to enable cheaper inference.
Also the opportunity to play around with all the parameters fairly cheaply to find improvements. The todo section of the readme gives a small taste of that. Making bigger models works for OpenAI, but maybe the rest of us manage to make small models just perform better instead.
