My personal belief is that it is a significant effect across all disciplines, but that the objective nature of some disciplines offers occasional ways to combat it.
My own academia experiences were in machine learning and image processing, and even there it was a heavy competition for demo ware. All ideas were basically vetted for their plausibility of leading to a TED talk or a conference keynote demo. If you proposed something or showed passion for something, but it wasn't clear that you could talk it about it while wearing a black turtleneck, then it got rejected pretty quickly. Little if any time was ever given to whether the fundamental questions were good ones or whether their answers constituted value-adding directions for humanity to move in. And don't even get me started on teaching. I once got yelled at by an adviser because I spent the time to write up my recitation notes for my probability students in TeX, instead of just scanning my unreadable hand-written notes. Good luck convincing them that investing time into how to teach something is worthwhile.
The process of moving past qualifying exams did include some objective hurdles of study (sort of quality control for the university's or department's brand) but it largely consisted of convincing certain key status figures that you were On Their Team and would be a good spokesperson for the sorts of positions and ideas they most wanted to affiliate with.
It will surely vary by discipline, geography, political climate, etc. But I think this is the dominant reason why we have academics at all. I do not think that the well-worn stories about funding fundamental research for society's benefit are even remotely true.
I can assure you that many others are toiling away in areas that could not plausibly lead to a TED talk. It's not all show-and-tell prestidigitation, but some areas are heavier in that regard than others -- like machine learning, for instance.
Your response is rational, however, in its critique of many fields and forums. I'd like you not, however, to denigrate the incredible sacrifices being made by researchers in all fields of study. Of course, the snake-oil rules the day at the moment, but those are just the most vocal and observable of workers, and they cast the rest in an unfortunate and undeserved light. The startup world, in many ways, shares a large responsibility for this perniciousness, as institutions all vie with press releases for investment and donation money more now than ever before with the goal of reaping short-term profits in the marketplace.
While some fields have higher-end media outlets like TED talks or infotainment-style demoware, or like Richard Thaler's role in the recent movie The Big Short, other fields might have less sensationalist outlets like a Senate subcommittee hearing, a scientific press release, or inclusion in a documentary film or important literary outlet -- and any of these can lead to follow-on speaking engagements or invitations as an academic adviser or dignitary in various circumstances. And even in the case of Senate hearings, it's often not about accuracy or objectivity, but instead about Supporting The Correct Team.
The point is not what is the maximum media exposure of a given field. The point is that in any field, there is a small elite class who mostly controls (as cultural gatekeepers) the ability for lower-status and aspiring researchers to advance, much like party politics. They form networks of editorial boards, committees for awarding tenure, and have nebulous connections through upper management of academia, corporate boards, endowments, politics, and celebrities. Yes it can vary by field, but you see the same pattern either at a big scale or a smaller scale.
I certainly believe many researchers enter the research profession with pure motives. But it is the ones who quickly learn to sublimate their desire for intellectually rigorous research and social betterment in favor of political skills who naturally rise to the gatekeeper positions.
Sadly, the pure and highly impressive research labor of a lot of these lower-status academics is just pure waste. I don't mean to denigrate the academics for this -- their motives are pure; they are just operating in a rigged system where the only means to significant success is to "wise up" and realize that the best "product" you can provide is to sort of auction off the ability to affiliate with you by sublimating yourself to someone's coalition in exchange for resources and notoriety provided by that coalition.
> it is the ones who quickly learn to sublimate their desire for intellectually rigorous research and social betterment in favor of political skills who naturally rise to the gatekeeper positions.
Nevermind just academia. Sadly, what you've just described neatly sums up modern work life in most organizations (companies, non-profits, open source, etc.) that I've been a part of.
Startups may sometimes be an exception to this rule, but they often come with their own set of problems (frat house atmosphere, ageism, zero diversity, expectations of 80+ hour weeks, etc.).
> I do not think that the well-worn stories about funding fundamental research for society's benefit are even remotely true.
Your observations of academic politics mirror my own, but I don't think your conclusion follows. In general, academics don't have nearly enough ROI as political sockpuppets, documentary hosts, or sources of cocktail-party quips to justify the level of funding they receive. If those were the only outputs of the scientific process that society valued, society would have switched to more efficient sources of them long ago. Instead, there is a consistent perception among those who finance science (taxpayers and, to a much lesser extent, wealthy patrons) that it fills a valuable niche of human progress that isn't served by other institutions.
And they're right. Yes, society doesn't know how to align scientific incentives with actual scientific value, which means that large amounts of funding are wasted and that actual progress is largely incidental to the games that researchers play on a day-to-day basis. Is that any different from any other part of the economy? Incidental progress is still progress, and the scientific game is better at producing its variety of incidental progress than any other game in town, so the decision to fund it would still be rational even if those outside of academia knew how thoroughly rotten it frequently gets. Not that they would be surprised -- academia does not have a monopoly on bullshit and politics, far from it. That's just how the world works, and most people are not foreign to the notion of sticking with something even if the absolute efficiency is dreadful.
You're reading intent into the incentive landscape that society presents to science, arguing that it rewards X, Y, and Z so therefore X, Y, and Z must be what it really wants. I think ineptitude is a much likelier explanation. Society rewards X, Y, and Z even though it really wants A, B, and X. Promoting human progress isn't a "well-worn story," it's the reason why we bother at all.
EDIT: Also, it's worth keeping in mind that Machine Learning is currently a hot field and therefore attracts opportunists and scumbags in droves. They exist elsewhere too, but they're going to be over-represented in your corner of academia.
I think there is another aspect that makes determining scientific value difficult.
In the hard sciences there are a lot of esoteric sub-disciplines that do rigorous work that seems to be of little consequence. I would suggest that one of the costs of maintaining a technological society is maintaining practitioners in many of these sub-disciplines just to keep the thread of knowledge alive.
If it was all left to the private sector, such researchers would be out of a job as soon as they were not longer necessary to the tasks at hand.
So many times in my own research have I needed a particular bit of information, and I search the literature and I find that someone did collect the data 30 years ago, presented in a paper with 5 citations. It would not be tractable for me to collect that data myself.
The problems with current management of scientific productivity is that the management ideas employed are good for running a production line cranking out a B-24 Liberator per hour, which is not really the same thing.
> In the hard sciences there are a lot of esoteric sub-disciplines that do rigorous work that seems to be of little consequence.
I think people systematically under-estimate the value of all those small seemingly-inconsequential tweaks, variations, and improvements. We all prefer simpler narratives that "person X invented Y for the very first time in year Z and that was good."
Many times when you break down a world-changing invention or innovation, it's actually an older idea that is finally possible due to thousands of incremental improvements being turned to the combined use.
Even that understates the importance of "trivial" research. Sometimes research projects don't work out - if you have a success rate of 100%, you're not doing research. Moreover, the minor research projects are your way of practicing, getting to know the literature, and getting to know what not to do (the Edison quote applies here) when you're doing research. If all of your research is of the trivial variety, you will not be doing research for long, because you won't be getting grants and you won't have a tenure track position at a school that gives you time for research.
