The problem is, LLMs do not have a place to put "unknowns". You can train them to say "I don't know", but that doesn't come from a place of any sort of internal representation of the fact that they don't know. They don't know what they don't know, because they can't. There's nowhere for that information to be in their internal representations.
And that's because they are exactly what they say they are on the tin. They are Language Models. Why would we expect a Language Model to "know" things? And to especially know what they do not know? It's modelling Language, not Knowledge, and that is not just English word play, that is a true representation of the situation. They are called "Large Language Models" for good and real reasons.
The problem isn't on the LLM side. They're amazing technology. They just aren't amazing in the way we're trying to use them.
We need some sort of AI that is knowledge based, that uses LLMs to input and output text but is something other than an LLM. Human brains aren't just big LLMs either. I, of course, have no idea what that is. Identifying problems is much, much easier than identifying solutions.
Honestly, neural nets in general are structurally disinclined to ever say "I don't know" just by the way they work. Only a small percentage of humans ever really learn how to say "I don't know" in any significant context to any degree; we have a pretty big problem with that ourselves!
LLMs are not "AI"; that is, they are not all of AI, the only thing in AI, or the the final conclusion of AI. They're just a step. And as I've said before, I suspect that once we have the next step in AI, what we're going to find is that the correct amount of resources to dedicate to the LLM portion is actually a great deal smaller. LLMs are probably super hypertrophied for what they actually should be doing and I expect 2040 to look back at our era of thinking that LLMs alone could be "AI" in much the same way we look at the idea that everyone in the world would have a Segway or something.
And that's because they are exactly what they say they are on the tin. They are Language Models. Why would we expect a Language Model to "know" things? And to especially know what they do not know? It's modelling Language, not Knowledge, and that is not just English word play, that is a true representation of the situation. They are called "Large Language Models" for good and real reasons.
The problem isn't on the LLM side. They're amazing technology. They just aren't amazing in the way we're trying to use them.
We need some sort of AI that is knowledge based, that uses LLMs to input and output text but is something other than an LLM. Human brains aren't just big LLMs either. I, of course, have no idea what that is. Identifying problems is much, much easier than identifying solutions.
Honestly, neural nets in general are structurally disinclined to ever say "I don't know" just by the way they work. Only a small percentage of humans ever really learn how to say "I don't know" in any significant context to any degree; we have a pretty big problem with that ourselves!
LLMs are not "AI"; that is, they are not all of AI, the only thing in AI, or the the final conclusion of AI. They're just a step. And as I've said before, I suspect that once we have the next step in AI, what we're going to find is that the correct amount of resources to dedicate to the LLM portion is actually a great deal smaller. LLMs are probably super hypertrophied for what they actually should be doing and I expect 2040 to look back at our era of thinking that LLMs alone could be "AI" in much the same way we look at the idea that everyone in the world would have a Segway or something.