Regarding the number of researchers, of all of my friends who got a PhD only a single one did it because he liked research. For most it was a stepping stone for a totally unrelated career in industry and for some just the easiest option due to lack of an alternative.
Given that, is is surprising that progress does not scale proportionally? Maybe, maybe not. For sure my comment is not backed by anything but anecdotes.
> Regarding the number of researchers, of all of my friends who got a PhD only a single one did it because he liked research.
This is sad; luckily for me the percentage is higher among my friends. I suspect your experience is closer to the statistical mean (perhaps because there's a good chance your friends are a few decades younger than me and many of my friends).
Also: I bailed out because I didn't like research, or rather because I very like the practice of research but only ever worked on super obscure things, and I wanted to make a difference I could see. Also I didn't want to have to deal with the culture of research funding and related bureaucracy. Not that there's anything wrong with those who do like that stuff! It just seemed not worth getting a doctorate; it definitely would have made some things easier but in retrospect I made the right call (for me!).
There is a lot of basic research done that does make a difference but where that difference is unknown to the original researcher -- perhaps separated too far in time. I have found value in decades-old PhD theses in the life sciences, mainly physiology.
> There is a lot of basic research done that does make a difference but where that difference is unknown to the original researcher -- perhaps separated too far in time. I have found value in decades-old PhD theses in the life sciences, mainly physiology.
Have you ever tried tracking down the author to express your appreciation? It might make their day (or month!).
That's actually the system working as intended. There's not nearly enough faculty positions or even non-faculty research staff positions to accommodate all the PhDs who graduate. The presumption is that most people who graduate will go into something other than academia. If you want to see the system failing, look to humanities, where most people who get a PhD are legitimately interested in academics and yet they will, in the best case, be stuck as terminal adjuncts
Given that, is is surprising that progress does not scale proportionally? Maybe, maybe not. For sure my comment is not backed by anything but anecdotes.