> I do not think that the well-worn stories about funding fundamental research for society's benefit are even remotely true.
Your observations of academic politics mirror my own, but I don't think your conclusion follows. In general, academics don't have nearly enough ROI as political sockpuppets, documentary hosts, or sources of cocktail-party quips to justify the level of funding they receive. If those were the only outputs of the scientific process that society valued, society would have switched to more efficient sources of them long ago. Instead, there is a consistent perception among those who finance science (taxpayers and, to a much lesser extent, wealthy patrons) that it fills a valuable niche of human progress that isn't served by other institutions.
And they're right. Yes, society doesn't know how to align scientific incentives with actual scientific value, which means that large amounts of funding are wasted and that actual progress is largely incidental to the games that researchers play on a day-to-day basis. Is that any different from any other part of the economy? Incidental progress is still progress, and the scientific game is better at producing its variety of incidental progress than any other game in town, so the decision to fund it would still be rational even if those outside of academia knew how thoroughly rotten it frequently gets. Not that they would be surprised -- academia does not have a monopoly on bullshit and politics, far from it. That's just how the world works, and most people are not foreign to the notion of sticking with something even if the absolute efficiency is dreadful.
You're reading intent into the incentive landscape that society presents to science, arguing that it rewards X, Y, and Z so therefore X, Y, and Z must be what it really wants. I think ineptitude is a much likelier explanation. Society rewards X, Y, and Z even though it really wants A, B, and X. Promoting human progress isn't a "well-worn story," it's the reason why we bother at all.
EDIT: Also, it's worth keeping in mind that Machine Learning is currently a hot field and therefore attracts opportunists and scumbags in droves. They exist elsewhere too, but they're going to be over-represented in your corner of academia.
I think there is another aspect that makes determining scientific value difficult.
In the hard sciences there are a lot of esoteric sub-disciplines that do rigorous work that seems to be of little consequence. I would suggest that one of the costs of maintaining a technological society is maintaining practitioners in many of these sub-disciplines just to keep the thread of knowledge alive.
If it was all left to the private sector, such researchers would be out of a job as soon as they were not longer necessary to the tasks at hand.
So many times in my own research have I needed a particular bit of information, and I search the literature and I find that someone did collect the data 30 years ago, presented in a paper with 5 citations. It would not be tractable for me to collect that data myself.
The problems with current management of scientific productivity is that the management ideas employed are good for running a production line cranking out a B-24 Liberator per hour, which is not really the same thing.
> In the hard sciences there are a lot of esoteric sub-disciplines that do rigorous work that seems to be of little consequence.
I think people systematically under-estimate the value of all those small seemingly-inconsequential tweaks, variations, and improvements. We all prefer simpler narratives that "person X invented Y for the very first time in year Z and that was good."
Many times when you break down a world-changing invention or innovation, it's actually an older idea that is finally possible due to thousands of incremental improvements being turned to the combined use.
Even that understates the importance of "trivial" research. Sometimes research projects don't work out - if you have a success rate of 100%, you're not doing research. Moreover, the minor research projects are your way of practicing, getting to know the literature, and getting to know what not to do (the Edison quote applies here) when you're doing research. If all of your research is of the trivial variety, you will not be doing research for long, because you won't be getting grants and you won't have a tenure track position at a school that gives you time for research.
Your observations of academic politics mirror my own, but I don't think your conclusion follows. In general, academics don't have nearly enough ROI as political sockpuppets, documentary hosts, or sources of cocktail-party quips to justify the level of funding they receive. If those were the only outputs of the scientific process that society valued, society would have switched to more efficient sources of them long ago. Instead, there is a consistent perception among those who finance science (taxpayers and, to a much lesser extent, wealthy patrons) that it fills a valuable niche of human progress that isn't served by other institutions.
And they're right. Yes, society doesn't know how to align scientific incentives with actual scientific value, which means that large amounts of funding are wasted and that actual progress is largely incidental to the games that researchers play on a day-to-day basis. Is that any different from any other part of the economy? Incidental progress is still progress, and the scientific game is better at producing its variety of incidental progress than any other game in town, so the decision to fund it would still be rational even if those outside of academia knew how thoroughly rotten it frequently gets. Not that they would be surprised -- academia does not have a monopoly on bullshit and politics, far from it. That's just how the world works, and most people are not foreign to the notion of sticking with something even if the absolute efficiency is dreadful.
You're reading intent into the incentive landscape that society presents to science, arguing that it rewards X, Y, and Z so therefore X, Y, and Z must be what it really wants. I think ineptitude is a much likelier explanation. Society rewards X, Y, and Z even though it really wants A, B, and X. Promoting human progress isn't a "well-worn story," it's the reason why we bother at all.
EDIT: Also, it's worth keeping in mind that Machine Learning is currently a hot field and therefore attracts opportunists and scumbags in droves. They exist elsewhere too, but they're going to be over-represented in your corner of academia.