I wonder what it would look like if you’re not allowed to cite papers in the field unless there’s been 5 independent labs that have managed to reproduce the result and all the labs that reproduce get to share in the credit (eg if there’s a Nobel prize). Would that change the incentive structure and create better outcomes? Science isn’t science until you have a repeatable process in place where observations match outcomes and the underlying theory. Someone proposing a bold new theory without the ability to test it usually gets significant credit even if posthumously for being a bold visionary which seems to have been sufficient throughout time as a motivator for the really foundational scientific victories.
The effort required to reproduce a result, especially an exotic one, is often prohibitive. The reward for successful reproduction is near zero, and if you fail, it’s often absolutely zero - you are not going to be able to contest a result by a failure. The original authors can simply say that your reproduction was flawed, if they need to answer at all.
A journal that requires five confirmations to cite a work is a journal that will fast have zero submissions.
The main reason people attempt reproduction is to continue that line of research, and that often starts with reproduction plus a slight tweak. This is why you find a lot of papers like this. This is also how irreproducible results become sort of known about but not challenged in many fields.
So what does the incentive structure look like? Do you get tenure for running a lab that replicates? Are your postdocs in line for tenure track positions? How is it funded: do you put the secondary replications into the original grant? What happens if you can't find X number of labs to run the test?
It's an interesting idea, but it all comes down to how the incentive structures happen.
