These anthropomorphizations are increasingly absurd. There's a difference between a human making a mistake, and an AI arbitrarily and completely confidently creating entirely new code APIs, legal cases, or whatever that have absolutely no basis in reality whatsoever, beyond being what it thinks would be an appropriate next token based on what you're searching for. These error modes are simply in no way, whatsoever, comparable.
And then you tell it such an API/case/etc doesn't exist. And it'll immediately acknowledge its mistake, and ensure it will work to avoid such in the future. And then literally the next sentence in the conversation it's back to inventing the same nonsense again. This is not like a human because even with the most idiotic human there's an at least general trend to move forward - LLMs are just coasting back on forth based on their preexisting training with absolutely zero ability to move forward until somebody gives them a training set to coast back and forth on, and repeat.
I mean I can definitely remember lots of cases for myself, in school especially, when I made the same mistake again repeatedly despite being corrected every time. I'm sure today's language models pale in comparison to your flawless genius, but you seriously underestimate the average person's idiocy.
Agreed that the lack of some mid tier memory is definitely a huge problem, and the current solutions that try to address that are very lacking. I highly doubt we won't find one in the coming years though.
It's not just this. LLMs can do nothing but predict the next token based on their training and current context window. You can try to do things like add 'fact databases' or whatever to stop them from saying so many absurd things, but the fact remains that the comparisons to human intelligence/learning remain completely inappropriate.
I think the most interesting thought experiment is to imagine an LLM trained on state of the art knowledge and technology at the dawn of humanity. Language didn't yet exist, slash 'em with the sharp part was cutting edge tech, and there was no entirely clear path forward. Yet we somehow went from that to putting a man on the Moon in what was basically a blink of the eye.
Yet the LLM? It's going to be stuck there basically unable to do anything, forever, until somebody gives it some new tokens to let it mix and match. Even if you tokenize the world to give it some sort of senses, it's going to be the exact same. Because no matter how much it tries to mix and match those tokens it's not going to be able to e.g. discover gravity.
It's the same reason why there are almost undoubtedly endless revolutionary and existence-altering discoveries ahead of us. Yet LLMs trained on essentially the entire written corpus of human knowledge? All they can do is provide basic mixing and matching of everything we already know, leaving it essentially frozen in time. Like we are as well currently, but we will break out. While the LLM will only move forward once we tell it what the next set of tokens to mix and match are.
Sure, it happens. How often it happens really depends on so many factors though.
For example, I have this setup where a model has some actions defined in its system prompt that it can output when appropriate to trigger actions, and the interesting bit is that initially I was using openhermes-mistral which is famous for its extreme attention to the system prompt, and it almost never made any mistakes when calling the definitions. Later I swapped it with llama-3 which is way smarter, but isn't tuned to be nearly as attentive and far more often likes to make up alternatives and don't get fuzzy matched properly. Someone anthropomorphizing it might say it lacks discipline.
And then you tell it such an API/case/etc doesn't exist. And it'll immediately acknowledge its mistake, and ensure it will work to avoid such in the future. And then literally the next sentence in the conversation it's back to inventing the same nonsense again. This is not like a human because even with the most idiotic human there's an at least general trend to move forward - LLMs are just coasting back on forth based on their preexisting training with absolutely zero ability to move forward until somebody gives them a training set to coast back and forth on, and repeat.