Replication is over-emphasised. Attempts to organise mass replications have struggled with basic problems like papers making numerous claims (which one do you replicate?), the question of whether you try to replicate the original methodology exactly or whether you try to answer the same question as the original paper (matters in cases where the methodology was bad), many papers making obvious low value findings (e.g. poor children do worse at school) and so on.
But the biggest problem is actually that large swathes of 'scientists' don't do experiments at all. You can't even replicate such papers because they exist purely in the realm of the theoretical. The theory often isn't even properly written down! They will tell you that the paper is just a summary of the real model, which is (at best) found in a giant pile of C or R on some github repo that contains a single commit. Try to replicate their model from the paper, there isn't enough detail to do so. Try to replicate from the code, all you're doing is pointlessly rewriting code that already exists (proves nothing). Try to re-derive their methodology from the original question and if you can't, they'll just reject your paper as illegitimate criticism and say it wasn't a real replication.
Having reviewed quite a lot of scientific papers in the past six years or so, the ones that were really problematic couldn't have been fixed with incentivized replication.
So then, how on earth does this stuff even get published? What exactly is it that we're all doing here?
If a finding either cannot be communicated enough for someone else to replicate it, or cannot be replicated because the method is shoddy, can we even call that science?
At some level I know that what I'm proposing isn't realistic because the majority of science is sloppy. P-hacking, lack of detail, bad writing, bad methods, code that doesn't compile, fraud. But maybe if we tried some version of this, it would cause a course correction. Reviewers, knowing that someone actually would attempt to replicate a paper at some point down the road, would be far more critical of ambiguity and lack of detail.
Papers that are not fit to be replicated in the future, whose claims cannot be tested independently, are actually not science at all. They are worth less than nothing because they take up air in the room, choking out actual progress.
That correct. Fundamentally the problem is foundations and government science budgets don't care. As long as voters or Bill Gates or whoever believes they're funding science and progress the money flows like water. There's no way to fix it short of voting in a government that totally defunds the science budget. Until then everyone benefits from unscientific behaviour.
The amazing thing is that it all works out in the end and science is still making (quite a lot of) progress.
That's also the reason why we shouldn't spend all of our time and money checking and replicating things just to make sure noone publishes fraudulent/shoddy results. (We should probably spend a little more time and money on that, but not as much more as some people here seem to suggest).
Most research is in retrospect useless nonsense. It's just impossible to tell in advance. There is no point in checking and replicating all of it. Results that are useful or important will be checked and replicated eventually. If they turn out to be wrong (which is still quite rare), a lot of effort is wasted. However, again, that's rare.
If the fraud/quality issues get worse (different from "featuring more frequently and prominently in the news"), eventually additional checks start to make sense and be worth it overall. I think quite a lot of progress is happening here already, with open data, code, pre-registration of studies, better statistical methods, etc, becoming more common.
I think a major issue is the idea that "papers are the incontestable scientific truth". Some people seem to think that's the goal, or that it used to be the case and fraud is changing that now, however, this was never the case and it's not at all the point of publishing research. I think a major gain would be to separate in the public perception the concepts, understanding and reputations of science vs. scientific publishing.
But the biggest problem is actually that large swathes of 'scientists' don't do experiments at all. You can't even replicate such papers because they exist purely in the realm of the theoretical. The theory often isn't even properly written down! They will tell you that the paper is just a summary of the real model, which is (at best) found in a giant pile of C or R on some github repo that contains a single commit. Try to replicate their model from the paper, there isn't enough detail to do so. Try to replicate from the code, all you're doing is pointlessly rewriting code that already exists (proves nothing). Try to re-derive their methodology from the original question and if you can't, they'll just reject your paper as illegitimate criticism and say it wasn't a real replication.
Having reviewed quite a lot of scientific papers in the past six years or so, the ones that were really problematic couldn't have been fixed with incentivized replication.