Here is something that I think would be a big breakthrough:
I explain to GPT in text, a mathematical concept it has never seen in its training data and give a few examples (not inferred from fill the blank on millions of examples). It actually learns this to update its weights - not just uses it as part of a prompt.
Extrapolating this optimistically - this is a huge step towards AGI in my opinion. You can (in theory) teach it to automate many tasks, correct it's mistakes without needing costly extra training data, and move towards the few-shot (and persistent) learning that separates humans from AI right now.
> It actually learns this to update its weights - not just uses it as part of a prompt.
In a way both are the same thing - memory that is in a feedback loop with the network that does the calculation. Just that the weights give much faster access, no "serde".
Maybe the goal is not to modify the weights but train the network so that it can effectively use a "memory block" in the way it works. Now this is in a way faked by re-feeding the output it produces concatenated with the original phrase. Don't we as humans effectively extend our memory by using all kind of text, written or digital? Just the issue is that it is slow to utilize, for a computer using fast RAM that wouldn't be much of an issue.
I explain to GPT in text, a mathematical concept it has never seen in its training data and give a few examples (not inferred from fill the blank on millions of examples). It actually learns this to update its weights - not just uses it as part of a prompt.
Extrapolating this optimistically - this is a huge step towards AGI in my opinion. You can (in theory) teach it to automate many tasks, correct it's mistakes without needing costly extra training data, and move towards the few-shot (and persistent) learning that separates humans from AI right now.