LLMs are not booksmart Harvard grads or anything else. They are extremely complex statistical models doing next-token prediction. That's it - that's all. If you want a proper mental model of LLMs, you need to understand this - the thing you're doing is text prediction. You do not have a partner engaged in cognition, you have a ludicrously complicated language model trained on a large corpus of data. If the bulk of that data happens to assert that the sky is blue, the model is likely, but not guaranteed, to finish the sentence "What color is the sky" with "Blue". That's it. That's the trick.
> They are extremely complex statistical models doing next-token prediction. That's it - that's all. If you want a proper mental model of LLMs, you need to understand this - the thing you're doing is text prediction.
Technically correct, but that's 100% what, 0% how - when the how is what 100% matters and the what doesn't matter at all.
Those models are able to coherently complete the next word, then next, then another in such a way a very useful word sequence is likely to appear which e.g. tells me how to do stuff with pandas dataframes, with working code examples, and then is able to tweak them (no small feat as anyone who can do that can attest). The only way to do that is to have some kind of smarts doing very non-trivial computations to arrive at the next-next-next-next...-next word that makes sense within the context of previous words and words that haven't been yet generated/sampled/statistically selected.
It does not need to think in the human sense to do that; proof is by demonstration.
> If the bulk of that data happens to assert that the sky is blue, the model is likely, but not guaranteed, to finish the sentence "What color is the sky" with "Blue". That's it. That's the trick.
Yeah. A very useful trick. And fiendlishly hard to learn. Perhaps there's a lot going on behind the scenes to make the trick work?
> The only way to do that is to have some kind of smarts doing very non-trivial computations to arrive at the next-next-next-next...-next word that makes sense within the context of previous words and words that haven't been yet generated/sampled/statistically selected.
Do you have any evidence for this claim? Like...I would like to believe this. I would like to say that LLMs are genuinely different in some way. But it just seems very possible that if you throw 10-100x the compute at "raw statistical correlation" it generates much better, specific sequences.
> Perhaps there's a lot going on behind the scenes to make the trick work?
> But it just seems very possible that if you throw 10-100x the compute at "raw statistical correlation" it generates much better, specific sequences.
This isn't just very possible, it's _what's happening_. They're fantastically complicated statistical machines. We know this - that part, at least, we _did_ write. The actual statistical model, we don't fully know, but we know what the machinery is.
There's an impression here of understanding sufficient that it warrants both further investigation and a lot of introspection with regards to the nature of intelligence, but these are not systems with any real autonomy or agency, and in particular the _way_ they're currently used and deployed almost precludes talking about things like understanding or knowledge in any real sense - they're run as one-shot, feed tokens in get token out forward-pass algorithms.
This is why I'm beating this drum - it's possible the algorithm that's being used underpins intelligence and understanding in a broad sense, it's possible (and some evidence suggests) the resulting models contain structures or patterns that encode information beyond the strict content of the input data, but when you interact with these systems today - like, in ChatGPT, Copilot, or anything else I'm aware of - you're inserting a string of tokens at the beginning of the model, receiving a predicted next token at the end of the model, and then repeating for however many tokens seem appropriate. This model is not talking to you, it does not understand what you asked, there is no growing mutual understanding, and assuming such will lead you to make assumptions about the behavior of the model that are incorrect.
We all know what the trick is, but none of us know why the trick actually works.
Especially the people now shouting that the models "do not understand", but instead are just doing next token prediction, would for sure not have foreseen that next token prediction apparently simulates understanding. I mean, they still don't see it, although it is there for everyone to try out.
First is humans are bad at grokking complex systems - simple rules applied at large scale can create enormously complex systems, and we're bad at predicting what those things are going to be.
The second is these systems are "self-tuned" - we explicitly designed the basic structure, but the interactions between different parts of the structure are derived, not created, and so we can't say exactly why those parameters are picked.
The final one is that they're simulating language, and we're a species constitutionally prone to anthropomorphizing - if something looks like it's acting with an intent we recognize, we assume it is. That's helpful in an evolutionary sense, but it means it's really easy to fool us into thinking something is sentient. We've been doing this to ourselves since Eliza.
The LLMs look impressive, but it takes very little time or effort to notice there's nobody home - not least of which is you need to keep feeding the model everything it's said before and if you tap out of the context window, it forgets. It's a very neat parlor trick, but there's nobody home there, at least not how they're currently constructed and run.
I think there's something being captured in how the models are created and trained, but at best, the run model does not allow for consciousness, because there's no continuity and no updating of the model over time.
“Doing next-token prediction” isn’t a contradiction of “understanding” any more than it would be for a SO/friend who can complete your sentences.
But it’s useful to remember that autocompletion of sequences is at the bottom of LLMs, and a hefty dose of RLHF of whatever the RLHF raters thought was good output.
I agree, it is not a contradiction. But usually those who bring up doing next-token prediction imply that it is a contradiction.
Also, there is often some confusion between consciousness and understanding. LLMs clearly understand on some level, within certain intrinsic and extrinsic constraints, but of course there is no consciousness there.
This model isn't especially helpful. "Text prediction" is correct, but glosses right over the fact that it's closer to "concept prediction", where a token is just a forced choice given the model's conceptual representation. Given the way the embedding layer learns, various tokens are likely to be chosen even if they've never been seen in the data in that sequence. This isn't a minor detail.
You can ask GPT4 to take some text and compress it to reduce its token count in a way that GPT4 can still reconstruct the meaning behind the text.
At no point was GPT4 trained on this, but it somehow has a level of meta-cognition that allows it to accomplish this task.
You can tell GPT4 it is an expert in writing GPT4 prompts and then ask GPT4 to write prompts for itself. (Granted now that the latest model is updated to 2023 data, GPT4's training corpus now surely includes blog posts on how to write GPT4 prompts, but the previous cutoff didn't!)
And don't underestimate token prediction, a very large amount of human perception relies on predicting the next input. It is why a large number of optical illusions work, and why various "written word" visual tricks work, such as how people don't notice duplicate words in a sentence (Our brains have a prediction for what words come next and the brain will throw out a certain # of small tokens that don't fit that prediction before an error is raised!)
Heck your brain does massive tokenization, compression, and error correction on incoming speech!
The big difference is we can logically reason what sequence of words come next. Though this requires a conscious effort and our innate biased are always working against us. One can always say that chain of thought reasoning is doing this but for me at least this seems like bootstrapping logical reasoning onto next token prediction and is not always guaranteed to work
> The big difference is we can logically reason what sequence of words come next.
A lot of our language skills, especially in our first language, are implicitly defined and we are not consciously aware of the rules until someone points the rules out to us.
> seems like bootstrapping logical reasoning onto next token prediction and is not always guaranteed to work
The most fascinating philosophy class I took was in logical thinking, where we had to use predicate logic to diagram out English sentences.
The tl;dr of it is that towards the end of class we went over political speeches and diagrammed them out to see what was really being said/implied. In a lot of cases the speeches, some of which demonstrated an excellent and eloquent use of the English language, said literally, absolutely nothing at all. Listeners could easily find a lot of meaning in the words (by design!) but in fact no meaning was present at all.
The key here is that listeners would apply logic to the words, even where no logic existed.
I'm not saying GPT4 has human intelligence, but I am saying that we shouldn't think our brains, which are full of ugly hacks and work arounds, are operating on some higher plain of existence. Intelligence barely works, and it is trivial to bypass, trick, and lead astray the "thoughtful" part of the human brain.
Saying "GPT hallucinates facts" isn't saying much. The way memory retrieval works for people is literally "get a sketch of things that happened and use past experiences to fill in the details."
And it works so much better than literally everything else we've ever tried for language processing that all of us, domain experts very much included, were caught completely by surprise.
What this all ultimately means? Too early to tell. But writing it off as a 'trick' isn't a helpful model to figure out why it's so shockingly effective in the first place.
this. SO MUCH THIS. I am so tired of reading through pseudo-scientific blurb from people claiming LLMs have "mental models". Especially when using retrieval-augmented generation (which futzes behaviour even more) and attributing human behaviours to a smarter autocomplete.
"Mental model" is a useful shorthand for referring to it. We say an A-star pathfinding algorithm "wants" to minimize its cost function. We can say this while understanding that it's a computer that doesn't have desires. But we say it because it's an easier way to communicate. We don't have to talk like lawyers all the time.
It’s not that LLMs have mental models. It’s that people have mental models of how various things work.
For example, many people have a wrong mental model of how an oven works. They think that if they set the oven for 350 F it heats with more energy than if they set the oven for 150 F. In reality it heats at exactly the same power, but it turns off when the internal temperature reaches the set rate.
For most use cases it does not matter that your mental model does not match how the oven works. But there are some not he cases where it does matter.
The article shares a mental model of how LLMs work that might help users use them better.
