Scientific fraud robs the community twice: the first time by wasting research funds on fraud, the second time because experiments with the appearance of success funnel more dollars to them, taking away funding from other areas that might look less sexy but might have yielded a breakthrough of they'd been pursued. Grant issuers should insist on some form of end-to-end third-party data custodianship to prevent tampering with data during analysis. Seems better than the many billions of $ being wasted on fraud and subsequent missed opportunity.
Not to mention all of the students who end up dropping out or having to take Masters instead of PhDs because their theses built on the fraud fail.
Just like we should place failed results (when due to wrong science, not bad skills) on an equal level with successful results, we should place failed thesis projects on equal level with successful thesis projects as having added to the general knowledge (again, when the failed projects demonstrate a falsified theory, not when they fail due to mistakes or inability on the part of the student).
Great comment. I would encourage all PhD students to bomb-proof their thesis topic and build it on as solid a foundation as possible. I chose perhaps a less flashy, relevant, and lucrative topic for this reason but I would rather finish than be staring down the barrel of a "my entire work has been based off a lie"...
When you have lots of people whose livelihoods depend on the gravy train, who can't be sure that what they are working on is fraudulent because they are so specialised, who would take that risk?
Its all about funding. And in the US basically all funding comes from the same source - government, military or corporations - what I think of as the governance system.
It’s very weird to me that science now equals vaccines and home-mandates, and it also equals advice given with little data. Science is way way bigger, and is an idea not a particular set of people or current beliefs. It also has brought untold value of taking us out of the dark ages but apparently that’s been normalized so much it’s no longer valued.
Dan Ariely and Francesca Gino were two of the most well-known behavioral economists. Hell, Ariely even published a a board game about it (as well as a bunch of popular books). They've both been accused of data manipulation this year.
My friends in the field are worried that the whole field will be tainted. It grew out of a controversial idea - that despite what past models demand, people don't consistently behave in a rational way. If the two biggest practitioners of a new discipline are outed as frauds, what does that do the the reputation of the discipline as a whole? Will people be skeptical of any behavioral hypothesis that bucks tradition because Ariely was accused of fraud?
And they are far from the only ones: Diederik Stapel, Brian Wansink, the list goes on and on. (I'm not listing the many names, also at R1 universities, whose verdicts are still "in process.")
What NPR liked to call "replication crisis" was a combination of junk science and blatant fraud.
The whole field (slightly more broadly, of social psychology) is badly damaged for at least a generation.
Pete Judo has some consumer-friendly youtube videos on the topic.
So many problems in science could be alleviated if we changed the culture and processes to multi replication. Your paper can only be accepted into a paper if an unrelated group can replicate the results with only the paper and associated submitted info.
This doesn’t scale. Sometimes only a small number of labs have the equipment needed to run an experiment, let alone the training on the equipment to do what needs to be done. This equipment can cost millions. Even if the results are correct, experimental science is very finicky by its very nature. Getting an experiment to work can take months.
Fine, let's just do it on those that scale. Not all science is done in high-end, expensive labs, and this type of control will help the young researchers who are publishing their first works to gain more credibility and develop healthy habits.
Sadly, as long as the tyranny of publishing exists, the other researchers will always prefer working on their own things than replicating someone else's not yet published experiment.
I'd say if no one can afford to, or knows how to replicate the thing you did. What point is there in it? And only because such an approach has downsides it's not clear it means the overall benefit of lessening the replication crisis and forcing scientists to provide enough information their work can be replicated at all would be a huge win. Based on people like https://www.youtube.com/@AppliedScience experience, 9 out of 10 times there is some crucial information missing in the papers they are trying to replicate.
Because sometimes they can afford to build on your prior results, and advance the state of the art. Factoring in all of the malfeasance, what's the trade off between not publishing due to inability/unwillingness to replicate, and publishing bad results? We should know this tradeoff before significantly changing the status quo.
Agreed - having done a PhD in an experimental neuroscience lab, there's a nonsignificant number of things that nobody else in the world, even in my own lab, can do. I can train the techniques to others, and do, but this is separate from scientific discovery. There are no incentives for someone else to spend their time replicating my work using unbelievably challenging and expensive methods. It just wouldn't work in practice without a more fundamental restructuring of the whole enterprise (which is also necessary but hard).
Like what? I can think of extremely few domains that any decent research university isn't well equipped to competently replicate. One of the very few benefits of the tuition explosion - even undergrads get relatively casual access to equipment worth millions of dollars.
Of course they can't replicate things like ultra high energy particle research, but these sort of obscure things make up a very negligible chunk of all science produced, even if it's quite an important little chunk.
I guess you're not thinking hard enough. I estimate >90% of research in most technical fields would require 10s of years to replicate unless you are one of the few labs already working on the same topic.
For example in on of the research areas I'm familiar with (optical communications), there are maybe 10 academic labs in Europe (and even less in the US) who have the equipment to reproduce some of our experiments. In our lab there is 1 PhD student who could pull off reproducing the more sophisticated experiments (because he is the one focusing on communications) it took him 2 years to get to that stage.
This is an relatively easy area, i.e. equipment is largely off the shelve, very applied with lots of industry involvement. There are plenty of experiments published which could only be done in 2 labs (both of them industrial), just due to the cost of the required equipment.
In other areas (e.g. with fabrication in the clean room) reproduction would require even more time investment.
Don't get me wrong, reproducing results is important, but what people don't realise it happens all the time when people do adopt part of published results into their research. Mandatory reproducing results would just create large overheads which would get us nowhere.
It's entirely possible there are fields with needs I am unaware of or not considering, but your response is not compelling and sounds like hand-waiving. Exactly what equipment are you talking about?
I suspect you are likely grossly underestimating the available supplies at many research university in the US. For instance things like class 100 clean rooms are basic facilities. Many (and I want to say most) research universities also have partnerships with (if not ownership) of various specialized labs in the surrounding areas for more specific purposes. For instance the NASA Jet Propulsion Lab is managed by Caltech.
Realtime oscilloscope at least 4 channels > 50 GHz bandwidth $0.5M
(for some research you need >=12 channels so multiply that number)
Arbitrary waveform generator 4 channels > 45 GHz bandwidth $0.3M
(again you might need more than for channels)
RF amplifiers, electro-optic components etc. easily cost $2000-$5000 each and you need several (4-8 at least) of these. The RF cables and connectors/adapters easily cost several thousand $ each.
Fibre components, subsystem components (e.g. a WSS of which you will likely need 4 or so is $50k).
And I will certainly not let a student without training touch the sensitive high-speed RF equipment.
Regarding your comment on clean-room. For many fabrication purposes class 100 is not sufficient (also calling it basic facility is quite rich). And the equipment in is very expensive, LPCVD machines, E-beam, other lithography ... is $10sM. Most universities I'm aware of require fees (typically paid from grants) of $10sk per year to use the facilities (those are the reduced rates for university staff). The training/certification on the equipment typically takes about 1year.
Regarding JPL, yes it's managed by Caltech, what do you think NASA will say if Caltech professors will ask for a student to use the facilities to verify some
paper? Sure, lets delay the next Mars mission a year or so, to let some PhD students try stuff in the labs.
I think you seriously underestimate what the cost of using all that equipment is and how much training is involved to be allowed to use it.
You definitely don't want any
When I say a class 100 cleanroom is a 'basic' facility, I mean it's one you'll find at any decent research university in the US, and it is. If there's something that can be reasonably expected to be required for cutting edge research, you'll find it. As for lab access, my experience is in CS. I was granted access to a globally ranked supercomputing system paired alongside a paired largescale audio-visual facility. The only requirements for access were to be whitelisted (involved in research) and registered. After that of course you needed to slot/reserve time, but it was otherwise freely and unconditionally available.
It's difficult to really explain how much money is spent in top US universities. It's as if there's a fear that revenues might manage to exceed costs. But one of the practical benefits of this is that bleeding edge hardware and supplies, at costs far greater than anything you've listed, is widely and readily available.
I think the cost and complexity of reproducing work is somewhat overestimated, as is the specific expertise of individual researchers, though maybe your field is exceptional in this regard.
Primary research, pioneering new techniques and equipment to explore the unknown, is time-consuming and costly and requires a lot of original thought and repeated failure until success is achieved. However, reproducing that work doesn't involve much of this. It's taking the developed methodology and repeating the original work. That may well involve expensive equipment and materials, and developing the technical expertise to use them, but that does not involve doing everything from scratch and should not take anything like as long or cost as much.
I also believe that we far too readily overestimate the specific special skills which PhD students and postdoctoral researchers possess. Their knowledge and skills could likely be transferred to others in fairly short order. This is done in industry routinely. A PhD student is learning to research from scratch; very little of their expertise will actually be unique, and the small bit that is unique is unlikely to be difficult for others to pick up. I know we don't like to think of researchers as replaceable cogs, but for the most part they are.
My background is life sciences, and some papers comprise years of work, particularly those involving clinical studies. However, the vast majority of research techniques are shared between labs, and most analytical equipment is off the shelf from vendors, even the very expensive stuff. Custom fabrication is common--we had our own workshop for custom mechanical and electronic parts--but most of that could have been handled by any contract fabricator given the drawings. And the really expensive equipment is often a shared departmental or institutional resource. Most of the work undertaken by most of the biological and medical research labs worldwide could be easily replicated by one of the others given the resources.
Depending upon the specific field, there are contract research organisations worldwide which could pick up a lot of this type of work. For life sciences, there are hundreds of CROs which could do this.
As one small bit of perspective. In my lab a PhD student worked on a problem (without success) for over a year. We gave it to a CRO and they had it done in a week. For less than £1000. The world is full of specialists who are extremely competent at doing work for other people, and they are often far more technically competent and efficient than academic researchers.
It's broadly true that most research is eventually subject to attempts at replication. However the replication isn't explicitly attempting to reproduce the exact prior research, but to build on it.
If the foundation established by the prior research is flawed, attempts to build on it will usually fail.
Some practical ideas, when allocating the money for research 30% or so has to be kept back for replication. Public science is mostly government funded. And the government could allocate X% percent of the budget for replication work. It sounds like a solvable problem to me.
I agree in principle but forcing institutions or governments to allocate money for this and enforce is pretty much an unsolvable problem at this point IMO.
