> One thing that is becoming more and more clear, is that memorization is absolutely a required part of becoming an expert.
Has this ever been in question?
I don't think any serious expert in ML that is pushing against LLMs is making a claim that memorization and/or compression isn't a necessary part of intelligence. Rather that there's more to it.
Too much memorization is a bad thing, it's called over fitting. Bringing up schools is a good example. I'm sure many here have met people who can answer questions really well when in specific contexts but not in others. People who do well on tests but not in the lab. The difficulty of word problems is a meme, but are just generalization.
If you ask me, what makes humans and animal brains special is the fuzziness. It's this seemingly contradictory nature of well defined understanding through rules (such as physics) but also understanding that resolution is far from perfect. In our quest to become more precise it is recognizing the impossibility of precision and finding balance.
What I'd say is wrong with both the US and China is failing to teach how to think. The truth is that this is exceptionally difficult to test, if not impossible. It's difficult to distinguish from memorization when the questions are not clearly novel. But how do you continually generate sufficiently novel questions when not teaching at bleeding edge?
> Too much memorization is a bad thing, it's called over fitting.
It's not. It's just a skill that might become unused, like playing piano. Children who grow up in religious cults that emphasize memorizing the holy texts (Hasidic Jews, some Muslim sects) do surprisingly well on standardized tests. Even though they receive a fraction of instruction time.
> What I'd say is wrong with both the US and China is failing to teach how to think.
And I maintain that you can't learn how to think without grinding through facts, learning how to organize them in your mind.
This article does not appear to be supporting a point counter to what I said. It is also focused on the opinions on non-experts. In fact, the majority of the article is discussing how schools aren't "following the data."
I would be surprised if the dominant method was "memorize" for reading, as this would mean a curriculum that has little free reading.
I'm not too interested in what non-experts have to say unless there is quite compelling evidence. The average person quire frequently overestimates their confidence in how something should be done.
> It's not. It's just a skill that might become unused, like playing piano
I grew up playing piano and memorizing holy texts. Even participating in scripture competitions as well as music competitions. With the highest confidence I can assure you that no professional in either of these subjects believes that one should memorize without limit. In music they will use the words "without soul" while in scriptures they may say that you know the words but not the meanings.
I'll directly quote from the bible to demonstrate both at once:
> Therefore I speak to them in parables, because seeing they do not see, and hearing they do not hear, nor do they understand.
Mathew 13:13[0]
I suggest reading the chapter in full, as it makes the point more explicitly.
I think both groups understand something important to language (which yes, I will argue that music is _a_ language): that the "words" (sounds) used are only tools to convey what is the deeper meaning inside. In music you seek to draw that out of the listener. In scriptures it is the same. The clearest cases of these may be proverbs, parables, koans, or fables. The words hold only what is at the surface. It is ironic you specifically mention Hasidic Jews, as they are deeply entrenched in the Kabbalah, which is famous for being entrenched mysticism. That there are hidden meanings in the scriptures. This isn't even uncommon in religion in general! I don't think it is hard to see this in music or any art. If you are in any doubt, please go visit your local art gallery and listen to one of the local artists. Even if you believe they are full of hogwash, it still illustrates that they are trying to convey something deeper. If you wish to get this lesson and learn a bit about Jewish mysticism at the same time I'd recommend A Serious Man[1] (a Coen brothers movie)
I must stress that language (of any form) has three key aspects: what is intended to be conveyed, the words and way the words are used (diction), and the way the person receiving interprets this. The goal is to align the first with the last, but there is clearly a lossy encoding and lossy decoding.
> I maintain that you can't learn how to think without grinding through facts
I'm not sure why you thought we were in disagreement. Perhaps you know the words but not the meaning. I hope your head does not feel too heavy from all the things you carry in your mind[2]
> This article does not appear to be supporting a point counter to what I said. It is also focused on the opinions on non-experts. In fact, the majority of the article is discussing how schools aren't "following the data."
In this case, "experts" who thought that "learning by playing" is better were wrong, as proven by data. Schools that use the traditional memorization-heavy approach of learning letter combinations do better.
The example is Oakland's schools, where teachers considered the traditional approach to be "colonizing". Test scores cratered as a result.
> I'm not too interested in what non-experts have to say unless there is quite compelling evidence. The average person quire frequently overestimates their confidence in how something should be done.
If you want to map the political system on more than one axis, you could look at economically liberal/conservative and socially liberal/conservative separately, but in a two-party system, not all quadrants will be equally represented. This particularly annoys libertarians.
If you want to map education politics on more than one axis, one axis could be progressive/conservative in the political sense (Are we 'woke'? Do we teach that some people are gay? Let kids pick their own pronouns? Do we have prayer in school? Base our values on the Bible? etc. etc.).
The other axis is 'progressive'/traditional in terms of methods, e.g. do facts matter and should kids learn by rote?
Again, not all quadrants are equally represented. The 'progressive' side in terms of methods genuinely holds that rote memorization is bad and useless, and even taeching kids to read with phonics is outdated - where that led, is explained in https://features.apmreports.org/sold-a-story/ which has been featured on HN before now too.
Unfortunately, most attempts to debate the methods axis online degenerate into fights about the political axis (see also: California math reform).
> What I'd say is wrong with both the US and China is failing to teach how to think.
The problem is, I'd say, that there is no such thing as "how to think". Thinking about environmental policy is very different from thinking about debugging **ing JavaScript callbacks (sorry, having a bad day). You need different degrees, for a start! There will never be a way to teach social scientists generically "how to think" so they can just pick up programming in 30 days, because all you need to do is apply your thinking skills to code! In the same way, we can't just "teach how to think" in a CS class and then expect the graduates to double up as MDs in a pinch, because that's just thinking about the human body!
You can only do "critical thinking" in an area where you have domain knowledge. That's one reason that memorization is required - you need the base of domain-specific facts so you have something to think with.
I don't think any serious expert in ML that is pushing against LLMs is making a claim that memorization and/or compression isn't a necessary part of intelligence. Rather that there's more to it.
Too much memorization is a bad thing, it's called over fitting. Bringing up schools is a good example. I'm sure many here have met people who can answer questions really well when in specific contexts but not in others. People who do well on tests but not in the lab. The difficulty of word problems is a meme, but are just generalization.
If you ask me, what makes humans and animal brains special is the fuzziness. It's this seemingly contradictory nature of well defined understanding through rules (such as physics) but also understanding that resolution is far from perfect. In our quest to become more precise it is recognizing the impossibility of precision and finding balance.
What I'd say is wrong with both the US and China is failing to teach how to think. The truth is that this is exceptionally difficult to test, if not impossible. It's difficult to distinguish from memorization when the questions are not clearly novel. But how do you continually generate sufficiently novel questions when not teaching at bleeding edge?