It feels like this is where training on code is going to go from important to cr...

mike_hearn on May 23, 2023 | parent | context | favorite | on: RWKV: Reinventing RNNs for the Transformer Era

It feels like this is where training on code is going to go from important to critical. Most human texts won't require you to look back 100,000 tokens to understand what it means, but if you dump 1000 source files one after the other than it will.