This is a nice toy application. Yay .. something other than MNIST! Is there a similar toy application for RNNs/LSTMs? I don't find the standard text generating application to be compelling. I've been looking for a simple dataset for teaching/learning purposes ... something like accelerometer or acoustic data. Haven't found a great one yet for recurrent networks.
For something simpler, starting from very shallow networks (and expanding them step by step), I wrote "Starting deep learning hands-on: image classification on CIFAR-10" https://blog.deepsense.ai/deep-learning-hands-on-image-class....