The main issue here is probably not squeezing memory but squeezing performance. ...

The main issue here is probably not squeezing memory but squeezing performance. Even using regular SIMD is not good enough if your network is medium sized. They apply linear quantization, lookups and special SIMD operations to make it speedy.

See here for what they did for offline speech recognition: http://static.googleusercontent.com/media/research.google.co...