I have a slightly inside opinion on this - before I became a Software Engineer, I got an MSc in a mostly unrelated field, and am a co-author on work published in a few (very middling) peer reviewed journals.
My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common. Researchers invest a huge amount of effort into a study, they really want the hypothesis to be true. A lot of studies have steps that are pretty susceptible to bias, like maybe you’re classifying results and there’s a bit of a judgement call there. It’s human nature to make biased decisions in these situations, and make enough of these biased decisions, and statistically insignificant results become statistically significant.
Some people might not see a difference between bias and fraud, but I personally do. I think of fraud as a very intentional deception, straight up falsifying numbers in a conscious attempt to deceive. While I see bias as more, you’ve got a borderline case, and you view it in the light you want to see it in, even somewhat unconsciously. Like the difference between unconscious racial bias, and overt hateful racism.
I think the best approach to combatting this is to spend a lot less of the overall $$ in science on novel research, and a lot more on attempting to independently reproduce results. Papers should be seen as meaningless until their results can be independently reproduced, and universities/colleges should reward reproduction studies as much as novel research. It’s kind of crazy that the system almost completely lacks these checks and balances right now - peer review is more like an editor, it’s just a very different thing than reproduction.
Bishop here is suggesting a data sleuthing approach to root out fraudsters, but I dunno if that’d be effective, as I think the main issue is subtle but pervasive bias seeping into studies by most researchers, vs. a smaller number of heavy fraudsters. Independent reproduction of results, while expensive, is the only effective approach I can think of to combat this.
> Some people might not see a difference between bias and fraud, but I personally do. I think of fraud as a very intentional deception, straight up falsifying numbers in a conscious attempt to deceive.
Lets say you do three runs of the same psychology experiment and only publish the results from the one where results are most significant. I think most researchers would not consider this fraud. You are publishing data that's absolutely true. It's the selective cherry picking of data in order to produce novel results that's rotting the field.
This is how I think researchers can sleep sound at night while faith in science continues to erode.
That I would consider fraud, personally. I’m referring to more subtle bias than that - like maybe you’re putting someone through a scenario, then trying to classify their emotional state afterwards. You classify the clear states properly, but are biased in classifying the borderline ones - like it’s not very clear if they’re sad or anxious, but if anxious fits your hypothesis better, you tend to classify these unclear states as anxious.
Maybe that’s not a good example, I’m not a psychologist. But in my experience, scientific studies are full of a surprising number of judgement calls, in all sorts of areas (experimental design, experimental execution, result classification, statistical analysis, exclusion of outliers, etc.), and it’s naive to think bias doesn’t seep into all of these, even for people who are quite committed to the ideals of science. I can’t think of a good way to combat this except very extensive independent reproduction of results.
Yeah, I never worked in a field that did double blind studies - did my MSc in Oceanography, and these were areas I felt bias could (and likely generally did) seep in within Oceanography. I assume each field has its own weak points, but I’d assume most fields do have them, since there’s so little real, independent reproducing of results, we’re basically depending on individual scientists to be perfectly objective, which just isn’t how human beings work IMO.
You're actually thinking of a triple blind trial: i.e. blinding the experimental subject, the scientists administering the experiment, and the statisticians interpreting the results. Double-blind is only the first two.
Interestingly, this correlates with fraud in journalism as well. One common technique is that there's a set of events that could be written about / published, but they choose to only write about the events that positively promotes their ideology, while the events that might negatively impact their ideology are ignored and not shared to the world. It's one of the ways that a news organization can "lie" without ever actually telling a lie. By painting a picture of the world using selectively chosen data based on a bias/agenda, rather than a picture of the world that is based on all available data.
“The first principle is that you must not fool yourself—and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists. You just have to be honest in a conventional way after that.”
— Richard Feynman
Then, in order to not fool yourself, you have to know your own weaknesses:
Poor science journalism. The joke that depending on the day of the week coffee is 'scientifically' either the best or worst thing in the world is what's killing trust in science. Most people aren't reading or hearing what scientists say directly, they're hearing cut up overly summarized tidbits or reading a clickbaiting journalist's interpretation of a paper that most of the time the journalists don't even consider to be worth linking to.
Notably, public university press departments are scarcely any better in this regard than for-profit popular science publications. These are journalists who should have direct access to the researchers and they still get stuff laughably wrong.
Post modernism. There’s a well established field of thought going past to at least the 60s that believes truth is relative, knowledge is subservient to language and therefore science is inherently biased and a tool of “power”.
It’s extremely popular in the west, especially among the elite and in the universities
As someone working at a university, it's definitely not the dominant ideology amongst the decision makers. Maybe in some humanities departments, but not in the rest of the university, and especially not in the administration.
Science had a religious like following, doctors were considered infallible, and so on
The fact that it is a well established notion that pharmaceutical companies buy studies is a new phenomena (relatively to before post modernism set in)
The extreme manifestations is anti-vax, and it’s usually not due to them coming out of a humanities department
There are many reasons, including continuous attacks by politicians on scientists and the scientific endeavour, inaccessibility of modern research to the average person, (seemingly) diminishing returns from scientific research, to name just a few.
Well scientists are shitty. For example, the entire masking bullshit could have been avoided if the science policymakers just said "hey we don't have any evidence, just dont be an asshole and put a mask on".
That's not what happened, and they discredited themselves. Guess what. The public does know what the scientific method is, and they smelled a rat.
Why should we trust scientists if they violate their own principles?
> continuous attacks
Yeah. These are often warranted. Scientists MUST be held to a higher standard because (unless they're funded by HHMI) they're using the public purse.
> My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common
There's also a grey area between fraud and not checking your work as thoroughly as you should.
For instance, I've witnessed a highly regarded researcher (Turing award) telling his co-author not to bother about some proof because nobody would read it.
Feynman has a bit where empirixal measurements of some universal constant (electron charge?) slowly crept up to the now-accepted true value. He said scientists are a little bit ashamed of this, because it showed that higher estimates were discarded and not published - bias.
BTW scientists get no upvotes for merely reproducing "known" results.
Is there a way to see natural merit/discovery in reproducing (not an artificial incentive, like grants for repeating studies). Perhaps analogous to how teaching helps your own understanding?
>My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common.
I have a similar experience and opinion (started a PhD, quit when I lost faith in the institution).
In our field in particular, it seemed to be an open secret that our current paradigm was a dead end, and this had been apparent for almost a decade by the time I joined the university.
Yet, in spite of this fact that we were going nowhere, there was extreme pressure to continue to toe the line. You couldn't just try something new based on a hunch, but instead had to perform fruitless experiments that were "guided by the literature" and therefore easier to justify to the people funding your endeavour.
The whole institution of academia is rotten, and going on a witch hunt like the article suggests ignores several massive elephants in the room.
Replication is kind of fuzzy though. A lot of things are replicated, just indirectly, building on suppositions supported by but not directly reproducing the study.
The problem here is perhaps one of reporting negative results.
It'd be very easy to rationalize that a secondary study didn't work because, hey it's not direct reproduction and this use case is invalid or the equipment available wasn't the same and etc...). And between the rationalizations and the lack of incentive to submit and publish negative results it's easy to extrapolate that an indirect challenge may take a really long time to be put forward.
Something kin to a bug report section might be in order. A "Works on my machine." regime isn't acceptable in the empirical domain. This would help prune dead ends in both primary and secondary+ research, one would hope, without necessitating direct redundancy. Of course this may run against the grain, fears of getting scooped or IP challenges and etc...
One of the first things a professor taught me on a class on the design of experiments is "Never doom yourself to success."
The experiment shouldn't inherently be designed to confirm your biases. I've carried that with me ever since. I agree with you - I suspect bias, judgement calls that favor what you want, looking for errors intensively when you get weird results and not when you get expected ones, etc. are far more common than outright fraud.
Starting point of the article appears to be that this gut feeling you mention, that there is very little outright fraud, is not completely right. It appears to me that he claims that there is more outright fraud than we believe.
I can sympathize with the idea of a "scientific police force" looking for fraud, but the fact is that we already attempt the appearance of that through peer review.
The problem is that peer review right now only provides the appearance of oversight. Referees are not given the time, money, or resources to really dig into papers and look for issues. I have peer reviewed 4 papers this year, averaging around 3 hours per review. The onus is largely on the referees/editors/publishers to immediately find something wrong in the paper to reject it rather than on the authors to really prove their case. And I mean "immediately". One journal that I review for requires peer reviews within 2 weeks of accepting the assignment, and they will nag you if you do not have it done within a week. If the onus must be on the referees, they need to be given much more time and resources, and that might even include money, and also might include waiting for reproduction of the results before publishing.
"Publish or perish" also plays a big role here, because scientists need to get papers published to prove they are productive and worthy of funding and employment. Authors will just re-submit their manuscripts to different journals until they are published rather than re-evaluating the research in any fundamental manner (is my methodology flawed? etc.). So putting the onus on referees isn't going to work in the first place, since peer reviews are not always shared.
Peer review’s fundamental flaw is that science only works through replication and peer review tries to sidestep that as a cost saving measure. It doesn’t mean there’s no value - having peers review your work can be valuable. But as any professional coder can tell you, a code review is an extremely low quality signal that can only catch obvious bugs, typos, and stylistic/basic rule conformance issues. It can’t explain why you made various design decisions and whether those were any good to begin with. I think its current role as gatekeeper is probably overall more harmful than helpful as it lends legitimacy it doesn’t have a right to (ie this paper is right) but I don’t have a better suggestion other than independent replication. Maybe more separation between who designs, conducts and analyses experiments with the analysis people being rewarded if there’s no result while the design and experiment people being paid if there is to set up a feedback loop?
This article puts the onus on publishers / the scientific publication process to catch and prevent fraud.
This is like asking merchants to catch and stop fraud and crime (e.g. selling alcohol to kids); it's in their best interest to not catch fraud and maximize their income.
The reason they do clamp down on underage drinking is because they'll get fined/arrested / their license will be revoked. The system cares enough to catch and punish the behavior.
If the funding bodies like NIH don't catch, punish and stop this behavior, it creates a system where fraudsters win more. This makes more groups, even if reluctant, participate in fraud because that's the only way to compete. It's a race to the bottom.
Money drives incentives, and clawing it back while blacklisting and publicly humiliating a lab, will change behavior. That would make the lab a toxic collaborator, especially if collaborators ALSO get blacklisted from funding.
NIH is using volunteers to review grants and some programs. So same problem applies. I've even seen reviewers stealing the work from grants they reviewed (and noted really badly so it wouldn't get funded) and that same reviewer did it multiple times. Nobody cared at NIH, the program officers said they would look into it and many years later still nothing happened. There is nothing really in place to solve blatant abuse.
What changes to the incentive structure do you think would help here? Blacklisting seems appropriate for repeat offenders but I'd be more interested in changing the incentives for funding to prevent fraud in the first place. The problem with blacklisting in my view is that many fraudsters who get away with it will just continue anyway since they do not "learn their lesson" until long after the damage is already done.
I think a lot of knowledge about what is fraud is already distributed among researchers in their own fields but can't be accessed because no one wants to say those kinds of things in public. The whole system of academia is shrouded in a fog because saying "X field is dead" or "Y research is fraud" is just not done.
But this is the same system that is unable to move off paid journals when the web has allowed ~0 cost publishing for decades. They can't effectively retract papers when the knowledge has failed to hold up. Like we have so many ways to organize, publish and review information on the web but they haven't adopted any of them.
Literally a clone of HN or Reddit with upvotes and comments could solve half the problems with publishing and peer review. Actually publishing your data and your scripts no matter how hacky they are can help replication. This is a problem that is technologically easy to at least attack and improve if not solve. But none of that has happened so i don't have much hope, for some reason they aren't able to change.
It is a decent idea, but it can be really hard to spot scientific fraud. As in REALLY, REALLY hard.
Some fields, like machine learning, gets deluged with papers coming from all kinds of researchers - there's no unified scientific notation, so to speak. In one paper you could have a pure math authors, in some other you could have econ authors, and then maybe a theoretical physics author. All these could use wildly different notation to describe related ideas, or discuss ideas to solve problems in their respective fields.
And in that case, how do you detect bogus science? You could be a leading ML scientist, but not have the faintest idea about the other domain. Or the inverse - be a leading domain expert, but have little ML knowledge.
Not to mention, the people that are publishing fraudulent scientific papers aren't morons - they're likely publishing that kind of stuff because of bad incentives. Maybe you'll catch the laziest, least capable fraudsters - but trying to catch savvy researchers sounds like a daunting task.
In my mind, this type of task/job is more fit for full-time scientific "auditors".
> To date, the response of the scientific establishment has been wholly inadequate. There is little attempt to proactively check for fraud: science is still regarded as a gentlemanly pursuit
There is no incentive mechanism, it has been the problem all along. The peer review process is clearly inadequate or not up to the task in its current form.
Speaking as a former MSc student in CS, a lot of ML related roles are asking for paper publications especially in top journals. This is just more bad incentives for the students to hype up their paper even if it is good but not good enough for a top journal
> The president of Stanford University has resigned after an investigation opened by the board of trustees found several academic reports he authored contained manipulated data.
> Marc Tessier-Lavigne, who has spent seven years as president, authored 12 reports that contained falsified information, including lab panels that had been stitched together, panel backgrounds that were digitally altered and blot results taken from other research papers.
In the HN discussions about posts like this, there tend to be several comments from people saying they left the academic world because they were disgusted with what they saw. This is a worrying signal, because it tells me that fraud is so pervasive in some fields that just exposing one paper as fraud is not going to fix it. Also, like the article suggests, there appears to be pressure to dismiss the fraud suspicions by the institutions and the journals. These tell me that the problems are systematic, because the system does not automatically correct the faults -- but tries to prevent them from being corrected. In a culture that is based on trust, like science, we should see a very different attitude towards fixing failures.
Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.
My gut feel from what I saw is that outright fraud is rare, but bias seeping into the study is common. Researchers invest a huge amount of effort into a study, they really want the hypothesis to be true. A lot of studies have steps that are pretty susceptible to bias, like maybe you’re classifying results and there’s a bit of a judgement call there. It’s human nature to make biased decisions in these situations, and make enough of these biased decisions, and statistically insignificant results become statistically significant.
Some people might not see a difference between bias and fraud, but I personally do. I think of fraud as a very intentional deception, straight up falsifying numbers in a conscious attempt to deceive. While I see bias as more, you’ve got a borderline case, and you view it in the light you want to see it in, even somewhat unconsciously. Like the difference between unconscious racial bias, and overt hateful racism.
I think the best approach to combatting this is to spend a lot less of the overall $$ in science on novel research, and a lot more on attempting to independently reproduce results. Papers should be seen as meaningless until their results can be independently reproduced, and universities/colleges should reward reproduction studies as much as novel research. It’s kind of crazy that the system almost completely lacks these checks and balances right now - peer review is more like an editor, it’s just a very different thing than reproduction.
Bishop here is suggesting a data sleuthing approach to root out fraudsters, but I dunno if that’d be effective, as I think the main issue is subtle but pervasive bias seeping into studies by most researchers, vs. a smaller number of heavy fraudsters. Independent reproduction of results, while expensive, is the only effective approach I can think of to combat this.