If your memory doesn't work, what does it matter how much you have?
You can create ungrounded abstractions all you you want, but as an older person who has commiserated with other older people, the common conception that it's harder to learn and retain things in older age is a real thing
Sure, but what you're describing is the difficulty of learning as age sets in. That's very different from the original assertion- "If I start specializing elsewhere I'll forget the intricate details of the thing I'm already specialized in." - unless that person was describing themselves as older.
Perhaps fixing degradation will prove to be a less difficult task than increasing memory, as the growth of new neurons is still a very newly-studied phenomenon compared to treatment into neurodegenerative processes, and soon medical science will benefit the learning of all demographics.