I would expect reinjecting into the context repeats of the prior context would be considerably more disruptive. The language is primarily a syntax for specifying manipulations on prior context. It’s hard to imagine a tooling that could more concisely and unambiguously create the needed structure to refer to things existing in the prior context without repetition.
And I would note the instructions to manipulate the context are an important part of the context and output. This is just a highly concise and unambiguous way that primarily exists outside the latent semantic space, so I would expect attention would be high given the way it’s composed. I would have used more unique tokens to even more draw out the instruction language.
And I would note the instructions to manipulate the context are an important part of the context and output. This is just a highly concise and unambiguous way that primarily exists outside the latent semantic space, so I would expect attention would be high given the way it’s composed. I would have used more unique tokens to even more draw out the instruction language.