This is not a critique of the (nice) study, but what seems to me the overall context that should be kept in mind, especially once we are talking about "optimizing".
Optimizing "study, retention phase, test" for greatest knowledge retention at a delayed test time, is very different from optimizing for greatest value of knowledge learned.
To optimize learning value, learn things that are immediately useful, you can immediately incorporate into learning something else, and ideally both.
The sooner and more you use something, the greater its value AND the greater your retention will be.
If you have to learn something valuable but with no short term use (how to handle a rare brain surgery complication), find a way to use it. Create an ongoing useful project that will revisit that knowledge during the "retention interval" (e.g. a concise summary of rare situations you need to handle, for you and others, that you can revisit and improve with additional and updated knowledge).
So optimize "topic choice", "topic progression", and "study, (optionally) test, use, use, use", for total value of learning.
"Use" is motivation, test, study review, and value realization put together.
I think a lot of topics are much less sequential the further you go. As an adult I spend most of my time repeating the fundamentals of my field, and learning a topic deeply as needed. For children it probably makes sense to cram the multiplication tables.
Basically all undergrad stuff is fundamentals and "optimized cramming" (we call it bulimic learning, because you stuff knowledge into you just in time to throw it back up in the test) means tons of people "achieve" degrees without being even dimly aware of the existence of the fundamentals afterwards. If you have a bachelor's degree in e.g. computer science, the idea of encoding data shouldn't be an unknown unknown to you.
I would think that the opportunities to immediately "use" multiplication, instead of just practice it for tests, or some future numerate citizenship, would be omnipresent.
If you don't use something after you learned it, you miss out on:
1. Learning how it is actually applied
2. Discovering how the knowledge is useful for you personally, in ways you may not expect if you don't actually experience using it
3. Deeper understanding and mastery of the knowledge
4. Much much much better retention
It is worth creating some immediate use for new knowledge, even the smallest possible useful or creative project, for better retention alone.
> learning a topic deeply as needed
That is the ultimate use-driven learning model.
As for non-sequential, I agree. The more we manage our own learning, the more it is a directed graph (i.e. prerequisites translate to many follow up paths), and eventually just graphs (many ways to order topics, and alternate combinations of prerequisites for each topic, in any complex area).
To be clear, in the context of this article, the alternative to drilling multiplication until you've memorized it is to space out the learning over a period of time. It's probably good to study for the test, because next week we're learning PEMDAS, so you better be solid on your multiplication.
"these days"?
The goal is to memorize single digit products and use the multi-column algorithms when you need to do more digits. To which days are you referring where kids would be called upon to memorize multidigit multiplications?
In the 1980s I had a trapper keeper with a 12x12 table printed on it but even my seven year old ass knew back then that everything >10 was wasteful to memorize. :P
20 years ago my kid still had to learn up to 12x20. The 12s are pretty handy for a bunch of reasons, and especially if you live in the USA. Using base 10 for the metric system was a mistake.
It encourages finding content from native sources, learning new words in context from materials that interest you rather than textbook content, and helps you build your own personal corpus of sentences as you encounter them in the wild. Will soon expand to more media types than just web and epub, to include YouTube/comics/HDMI input/game emulators etc
If you find flashcards from others, you also get easy tools for discovering more sentences from source materials that might interest you or from your own corpus. More coming for this and other word/kanji-level tracking analytics. Offline-first and privacy friendly.
This is great. Reminds me of the writings of Khatzumoto (All Japanese All The Time). On that note, do you know how to contact him? His site got deleted recently and I have an idea for restoring a static version.
Thanks! Yes I've read his stuff and basically all discourse online since ~2005 when I wrote my first web app version of Manabi. I was also heavily influenced by the Breaking into Japanese Literature book and similar ones which had translations of words and pages, sometimes additional explanations, for short stories from Akutagawa etc. (as I've included in the Rashomon EPUB in Manabi Reader)
I don't know how to contact Khaz sorry I've never talked w/ him. There's a subreddit dedicated to his approach r/ajatt so you might find leads there. I'd also check archive.org of course
If you try out my apps pls drop any feedback, easiest is Discord (chat or issue tracker) https://discord.gg/4aF9yuASzb Thanks - recently went full-time on this so looking to grow it quickly and have lots more on deck
[Mods: it might be helpful to tag this paper as written in 2007]
It seems to me this paper is bringing to light the idea of spaced repetition for learning and this has become quite popular in the productivity/learning culture of today (e.g. Ali Abdaal).
> Alternatively, mathematics textbooks could easily adopt a format that engenders spacing.
I tutor middle school students in mathematics and this is definitely being implemented in their textbooks! At the end of each chapter, there a is normal chapter review practice test followed by a "Cumulative Practice" which reviews topics from the previous chapters in the book. These are especially beneficial to my students as, like the paper highlights, it promotes long-term memory of those topics.
> For example, although computer-based instruction typically provides extensive retrieval practice and rapid feedback, it offers a currently unexploited opportunity to schedule study sessions in ways that optimize long-term retention.
There is an immensely popular software called Anki which implements exactly this "spaced repetition"-type protocol.
There is a comparable software that has a friendlier UX: https://mochi.cards/. It's basically Anki, if Anki were smoother. Does cost a tiny bit though.
Anki is very much... an expert's tool? I think there's quite a lot of opportunity (not necessarily monetarily :)) for SRS software with a good UX.
Anki's data model is rather strange, which partly relates to its flexibility but has some unexpected downsides and trap-doors.
I would love a slightly more opinionated SRS tool! (Maybe with a bit more UI polish, rather than rewriting the backend in Rust)
I don't see how anki is an experts tool...unless you want it to be.
You can simply dump everything into a single collection, use basic 'front' and 'back' cards...and review. Yes, it may not be 'optimal' depending on how you mark cards, but for most people, it's good enough. Better than nothing, and probably an improvement as an addition to whatever they are currently doing.
If you want expert features, you can. You can drill, tag, organise, create different templates....but none of that is really necessary, or particulary pushed on you as something you should or have to use.
This is a common complaint, but feels like a "delicate flower" argument. The UX is pretty spartan, but a fancy whiz bang UI isn't necessary to remember facts.
Not everyone needs all that, and the people that do (which is fine, btw), is a reflection of the person more than the tool.
Anki is only an expert's tool if you fall into the all too common trap of trying to optimize everything by microtweaking the settings.
I mean, sure, if that's your thing and you enjoy it, go for it. But it is not necessary, and (SM2 vs FSRS arguments aside), it works right out of the box with the default settings and algorithm.
Like anything, trying to get that last 10% optimization is WAY more work than getting the first 90%. But "using Anki" (in any way) is such a multiplier over NOT using it that just taking that step is enormous.
Quizlet has a spaced repetition mode you can use, you might need to pay for it? I used it a few years ago for some stuff I needed to memorize in school.
I push the new guys to fix project documentation once they’ve figured out a tricky bit. It helps solidify their knowledge, and helps us double check that they understood, and it’s something they can contribute when they still haven’t become part of the bus number on anything yet.
That all sounds reasonable and smart, but the real reason I do it is the Curse of Knowledge. People in a system can’t see it from the outside. They make assumptions, use opaque or even misleading jargon, and employ circular logic. The new guy doesn’t know the lingo, or the circular logic. Their explanation will make more sense to the next hire than anything I can say. And having it written down this way can also give me new perspective on the system. Maybe it doesn’t have to work this way.
Adding a new smart person to the team is one of those golden moments for a team, IMO. You get a tiny window of watching them struggle until the tribal team knowledge seeps into them by osmosis. During that time they don't yet know who to ask the questions to directly so they will post to team slack channels or the lead directly.
One must capitalize on this brief period because smart programmers are flexible and adaptable. Very quickly they will acclimatize themselves to the mess that surrounds them and they will become as blind to the deficiencies as the rest of the team.
Adjacent to this: ignore criticism as feedback, but treat questions as feedback.
People who don’t “get” the code ask questions that contain feedback they often don’t even register as feedback. By the time they distill it to an actual criticism, they’re often so wrapped up in the problem they can’t be constructive, or they present an XY problem. But if three people ask you the same question about your code? You have a design issue. Fix it. Fix it now.
That puts me in a weird relationship with FAQs. FAQs aren’t informational, they’re confessional. Here’s all the times I fucked up and won’t admit it. Let me explain why I am right and you are wrong.
Funny enough, many organizations will actually punish questions due to optics, noise, or RTFM (which fair enough do read it, but also don't presume the asker didn't just because you understand what you wrote. Being written down doesn't mean it's understandable by a new reader, or that it's optimally setup for discovery)
I tried to help some dude understand integration by parts, which I’m not sure I’d understood myself. I warned him. But it was late, he was desperate, and I was just going to go play computer games anyway.
I figured it out, but I’ve no idea if I got him sorted out for his final.
The professor is interrupted in the middle of his lecture by a student who asks about one of the previous week's problems:
— Could you please work an example for us?
— No problem!
The professor writes a problem statement, then scratches his head, tugs on his collar, rubs his chin, and at last scribbles a complicated integral on the board.
— There you go!
— Excuse me, professor, but could you please be a little more explicit?
— What do you want from me? I just did the problem 3 different ways.
Only getting 2 ways to solve it brings us back to russian jokes:
A soviet maths prof finds out he'd make more money as a labourer in the shipyards, so after more than a few carefully-distributed bottles of vodka he manages to change profession. A month or two working in the yards, and he sees a notice on the bulletin board offering a bonus to workers who sign up for a special "maths for the proletariat" evening school course.
Figuring it's easy money, he signs up, brings a novel, and sits in the back of the class reading instead paying attention. Then the prof calls him up to the brownboard, and asks him for the circumference of a circle.
Suddenly, our ex-prof blanks. Frantically he scribbles and scribbles on the brownboard, only to wind up with ... -2πr. No, that can't be right, but what went wrong?
Just when he's about to crack trying to figure out his error, he hears a friendly voice from one of the front-row students:
Someday you will die. Then you will die again when people forget you. Only the effect you’ve had on the world, including teaching others, who pass that along, outlives that second death.
So what’s this about efficiency? You should be more worried about effectiveness, not efficiency. Particularly in this profession.
What is consider efficient if it yields less best results?
I understand time constraints and end goals may prohibit the 'best' approach.
What I do not understand is how can you say something is more 'efficient' if the yield in understanding is less than what you would get with another method.
Hopefully this will clarify my thought process here:
If 'Best' is to teach others and requires 10,000 hours to yield 90-99% understanding.
In contrast, 'Efficient' method requires 2,500 hours to yield 30-40% understanding. However, there is diminishing returns meaning that doubling your hours to 5,000 does not return you with 60-80% understanding, rather maybe closer to 50-60% understanding. With 7,500 hours closer to 65-70% and 10,000 hours may around 75-89% understanding.
Here you've spent the same amount of time but did not achieve the same level of understanding. I think you may have a dynamic 'Best' vs 'Efficient' curve and to switch between those options to optimize maximizing your level of understanding in the least amount of time.
The problem I have with this argument is that you cannot consider "efficiency" in a vacuum. You need to have a metric against which to measure it.
Consider these two scenarios -
Goal: remember where to look up information when it comes up in $JOB
Metric: how much you remember, how quickly you find the info
Goal: discover new hyper-efficient method of training an AI (or insert popular ML topic here).
Metric: percent improvement vs current pubished best practice (deliberately vague)
Required understanding to make progress: "like a Ph.D. from Stanford"
Now you can possibly measure something.
The idea achievement of "90% understanding" is VERY topic dependent. Simple topic? Sure 100% understanding, I remembered the Latin names of all of the plants in my house. Complicated topic? The information for "100% understanding" might not even be written in the textbook - it probably includes things like seeing the interconnections between the topics and being able to apply them in slightly different contexts.
Make sure you read the studies so you know what they're talking about. In this area, I think summaries are frequently misleading. You have to know what the real evidence is that substantiates the claims. (I cannot tell you how many times I have looked at the evidence and just rolled my eyes - obviously not applicable in settings where I wanted it to be.)
Yes, but there are plenty of exceptions. I know brilliantly knowledgeable and extraordinary clever people that are so socially awkward that they can't explain virtually anything.
Good question and I think you're making a valid point.
I think overall on average they would do a decent job of explaining it in writing to equally brilliant people. Not so sure about "dumbing it down" for the "others".
For language acquisition: find a partner who speaks the other language and wishes to learn yours, then converse with each of you using your L2, falling back to L1 only when circumlocution fails or to make corrections.
(then how do you practise listening to your L2? Eavesdropping, ancillary encounters, news, movies, etc. give plenty of opportunity, but speaking requires you to make an extra effort.)
Sounds great but I was expecting more evidence. They talk about this shuffling method, which interleaves material as a way to provide spacing and reintroduce material. But it appears to just be their opinion that it will help improve retention. Also what is the deal with this "hypothetical interaction between ISI and RI"? Why not do enough experiments to actually plot it out? Anyone can graph out a hypothetical interaction.
Years ago there was a great extended comment here on HN about teaching a linear algebra course using, among other tactics, a similar spacing of the homework problems. He reported excellent results. I wish I remembered the username. Ben something?
I really appreciate HN. There aren't many public online forums were one could ask about a user comment made on a related post posted years ago and in less than 20 minutes get two different responses linking to the comment from 15 years earlier.
Perhaps it helps the topic was "increasing retention". :-)
This comment reminds me of a recurring nightmare I have about a math class I forgot to drop, never attended, and have to pass for to graduate. I’ve no idea if I pass it or not, since I always wake myself up by then.
This guy wouldn’t have helped me though, since I never attended the class.
I'd highly encourage anyone interested in the contents of this article/learning about learning to read through the submitter's blog: https://www.justinmath.com/blog/
I also recommend https://www.learningscientists.org/posters from scientists in the field, which covers additional scientifically-effective approaches that a course designer or more long term approach might take. For example, not only applying Spaced Repetition, Interleaving and Active Recall (all possible through automated spaced repetition apps based on simply input), but elements like Dual coding (related to "Varied practise") - mixing visual and other elements when learning (which requires effort to create them), and elaboration practise (like free recall). To the end of applying these - and other critical elements like focusing on the motivation of the learner and enabling them to understand what they are missing with progress & identified "blindspot" misconceptions (on their own incredibly powerful), I've been developing Revision.ai since before GPT-3, through a Psychology MSc.
Content is good but he is part of a paid math program and some articles and submissions appear to be indirect advertising/marketing if you think about it. (but it's effective and less forceful)
https://www.justinmath.com/why-is-the-edtech-industry-so-dam...
This is good stuff but I'll say that it isn't as comprehensive as all that. These studies and findings are almost entirely focused on simple recall knowledge (isolated facts, vocabulary). That's an important part of learning certainly and it informs research on learning higher order concepts but it's not the full story.
Just to name one example, folks might look into research on conceptual change theory (eg Chi or Posner). This theory helps explain why a concept like electricity is so challenging to learn. The reason, in brief, being that naive conceptions make a category error and think of electricity as a thing rather than a process. And this theory then informs instructional practice. Specifically, teachers should be aware of difficult concepts and should design activities that force students to confront the contradictions between their naive models and more accurate/complete ones.
Mickie Chi also has fascinations research on active learning (ICAP) and related work on the effectiveness of peer learning.
Thanks for your example of the category error here. Are there other cases though? For example, sometimes I doubt my understanding of gradient descent because it is very hard to implement in code on data and show error reducing over time (writing from scratch). But in some examples of a few nodes I can calculate it perfectly. I can do the Maths manually.
What error might I be making? For the future, is there a list of types of errors?
From a more applied angle, a book like "10 steps to complex learning" might be helpful.
I come from a similar cog psych background as the Bjork Lab, so am a big fan of their research, but books like 10 steps come from instructional design, which is a bit more focused on the big picture (designing a whole course vs individual mechanisms).
It is super unlikely that an algorithm that was made based on feels would turn out to be the most effective. Especially since it requires some learning and errors to even learn how to use it.
> Because people forget much of what they learn, students could benefit from learning strategies that provide long-lasting knowledge. Yet surprisingly little is known about how long-term retention is most efficiently achieved.
I've always thought that the real problem is information relevancy. People need practical uses for remembering something beyond synthetic bullshit exams. Efficient techniques are great, but nothing demotivates more than not having a reason to learn beyond being told that you have to.
Nobody has to reach for flashcards, extensive notes, or advanced techniques when trying to learn something they're actually interested in, it's retained almost immediately and effortlessly. There has to be some kind of subconscious gauge of information relevancy that physically controls the level of absorption, a sort of "learning rate" if you will.
I agree motivation to learn and maintaining interest is key, to effect the learning rate you mention. The enjoyment itself, although a crucifying word to use in discussions when teaching students considered the "best of the best", still matters and predominates over the effectiveness factor for most of the semester for students, at least. Because of just being "told to learn" it.
Basically 70% of the semester most students are not studying 40 hours - they are doing 30 hours of real work, and perhaps only 15 hours effectively. And for good reason: Nothing is bridging or rewarding them in a way that interests/motivates them, for courses where the interest isn't natural.
A bridge to motivate them would be ideal. In 2021 I started using GPT-3 to generate motivational "reasons to learn a concept" cards for my flashcard app, - Revision.ai - which you can read about here in the 3rd item: https://www.revision.ai/articles/20ThingsRevisionAIDoesForBe... - the reason we disabled them was simple: we could never quite time the cards right to help the student when they needed it. When the app is closed, they aren't motivated - they don't see them. Mid study session? Showing such cards (or AI generated examples) interrupted the flow [https://www.instagram.com/p/CVVlIuVg31W/]
We have also tried recommending relevant short/mid length youtube videos for visual/"breaks" from overwhelming learning. That did not boost student success either. I guess it doesn't address that you are still being told to do it, not naturally flowing into learning.
I welcome any technical or conceptual ideas you have to improve this and help increase interest amongst students. We have found that turning lecture slideshows into sets of exercises with clear visuals[https://www.instagram.com/p/C5ByftwiJ00/], breaking content up, and showing progress does motivate students to study more(and in a semi-related dissertation I wrote, possibly reduce Test Anxiety and tension). Please let me know of any ideas you have!
I disagree. Motivation will keep you reading the material, but that alone will not make you remember it that much better.
When you're motivated about something you can practice, you may remember more by doing it (which also acts a spaced repetition). But good luck trying that with, say, astrophysics or macroeconomics.
When motivated, you're also more likely to pick up other books on the subject, which is another form of spaced repetition.
I'd guess, that overlearning is so widespread not because it is beneficial for an individual, but as a way for a teacher to deal with a lot of students at once. To measure just the right amount of learning the teacher needs to work with each student individually, and give exercises for them based on a quality of their newly formed knowledge. But the system is mostly tuned for mass-education to amplify the impact of a teacher's work.
I’ll mention that I’ve gone full-time on an iOS/macOS tool for learning Japanese through reading called Manabi Reader: https://reader.manabi.io
It combines reading and flashcards, such that it tracks every word and kanji you read and learn in order to show you analytics on what you need to learn to be able to read something or achieve JLPT goals, and highlights unknown/learning words in texts. Next up for the flashcard part is to replace the SM2 algorithm with FSRS for the flashcards, as well as having flashcards get passively reviewed simply by reading content.
I also suspect people are missing out on speed of learning when reviewing flashcards one at a time for hours. Besides actively recalling reviewing flashcards passively while reading, I’ll experiment with other review techniques like seeing a page at a time of vocab / revealable answers. Our minds absorb at the periphery of our vision and scanning/inputting a bunch of information at once, too. I’m unconvinced flashcard UI is the final expression of forgetting curve research based learning apps
Also working on Reader features such as manga/pdf/youtube/game emulators, plus expanding to all languages.
A lot of what's on that page is just nonsense. The page has a very clear bias toward one method and writes all sorts of unscientific drivel as a result.
I'm not in a position to judge this author's work as a whole, as this is not my area of expertise.
I know this person as an author of "super memo" algorithms for spaced repetition learning, so yes, he most likely have a bias, probably even financial stake to some degree, although not sure, I don't follow him in any way, so don't know if this ever was or still is the case.
In the past, I've used some of his algorithms in particular, with quite interesting results - subjectively speaking, not in any scientific regime, And, obviously, those are just one of many. I also remembered this was his area of interest in general, so assumed his wiki to be of some interest in the context of discussed article.
Thanks for your critical opinion, it will contribute to my priors about his writing.
I enjoy browsing r/Anki from time to time, and that site is frequently mentioned there. Spaced repetition is definitely a real thing, but the rest of the SuperMemo site is a bit “crazy”. The author does have a “20 rules” for good card writing which is pretty good advice though. http://super-memory.com/articles/20rules.htm
One thing I’ve found when I spend most of a day trying to learn things, is that having a ~1.5 hour nap is pretty important. Not sure what goes on, but it feels like it turns the disconnected “fog” of information in my head into something that is a least loosely linked and can be vaguely recalled upon rather than completely forgotten.
Unfortunately I kinda need to plan these sorts of days ahead, because trying to nap past 2PM is a good way to fuck my nighttime sleep.
I'm pretty sure the current thought is that you ONLY "learn" during sleep, if you consider learning to be filtering, organizing, and throwing out vs keeping.
The time between study and testing (retention interval) is the most salient factor for students. Repeating material at intervals attenuates the forgetting curve somewhat, but not enough to generate passing scores. Cumulative exams only force students to study all material in a massed study session prior to a final exam.
I say this as someone who works on education and as a lifelong learner who has tried many ways of improving retention.
Real life usefulness, money, time investment for future use, learning prerequisites for the things you are interested in, etc. There are so many reasons...
Simple, but not sufficient since you still need to pair your interest with some methods of study and practice. Which methods of studying are more effective or efficient is the question they're researching.
Rohrer, D., & Pashler, H. (2007). Increasing Retention Without Increasing Study Time. Current Directions in Psychological Science, 16(4), 183-186. https://doi.org/10.1111/j.1467-8721.2007.00500.x
High IQ. A component of IQ is memory--working and long-term memory. There is a reason why doctors are smarter than average--you got to be smart to memorize all that stuff. I am skeptical that hacks make much of a difference. There is also a difference between recall and understanding. This is why speed reading courses are dubious because you're not really understanding things, but just recalling items in the text.
Optimizing "study, retention phase, test" for greatest knowledge retention at a delayed test time, is very different from optimizing for greatest value of knowledge learned.
To optimize learning value, learn things that are immediately useful, you can immediately incorporate into learning something else, and ideally both.
The sooner and more you use something, the greater its value AND the greater your retention will be.
If you have to learn something valuable but with no short term use (how to handle a rare brain surgery complication), find a way to use it. Create an ongoing useful project that will revisit that knowledge during the "retention interval" (e.g. a concise summary of rare situations you need to handle, for you and others, that you can revisit and improve with additional and updated knowledge).
So optimize "topic choice", "topic progression", and "study, (optionally) test, use, use, use", for total value of learning.
"Use" is motivation, test, study review, and value realization put together.