I’m not sure I understand how embedding or RAG help with the use case articulated. Those are techniques for either adjusting the semantic space or bringing context in via information retrieval. The use case here is iterative refinement of parts of the existing context without pasting it back into the context. Aside from time saving it’ll also help mitigate large context that contain lots of repetition issues.
It’s hard to imagine a way embedding or RAG could help you with “take response 5 and 8, combine them and distill into a summary, but with a more emphatic style”
Because you are impedding the model performance by doing a task that can be handled by external libraries in the context window. It's just inefficient while also influencing the output of the model... There are literally no positives on doing it OPs way.
So we should test out the performance, because with GPT funnily enough, it's possible that this type of prompt could bias it even better.
Because e.g. if you need it to answer in a very specific way, it might be helpful for the ChatBot to see what the other potential ways are so it would know to contrast even more.
Yes, it's influencing the output of the model, but it's unclear to me without extensive trying in which way, because it could be positive influence as well.
As for efficiency. It doesn't add that much in terms of magnitude if you already have long conversations going on.
Finally. Using external libraries and building out the things that you are mentioning will take time, trial and error. Just modifying the initial prompt is easy, and you can start to use it immediately.
So the main positive is just that it's likely faster and easier to implement compared to building out those systems.
I would expect reinjecting into the context repeats of the prior context would be considerably more disruptive. The language is primarily a syntax for specifying manipulations on prior context. It’s hard to imagine a tooling that could more concisely and unambiguously create the needed structure to refer to things existing in the prior context without repetition.
And I would note the instructions to manipulate the context are an important part of the context and output. This is just a highly concise and unambiguous way that primarily exists outside the latent semantic space, so I would expect attention would be high given the way it’s composed. I would have used more unique tokens to even more draw out the instruction language.
There are no absolutes, especially with LLMs. What if tooling altered output by 0.01 percentage, the general variance of output is 5 percent and to build alternative to that tooling would cost quite a bit of money and time?
It’s hard to imagine a way embedding or RAG could help you with “take response 5 and 8, combine them and distill into a summary, but with a more emphatic style”