We have the answer. Peer reviews should include a separate team replicating the results (follow methods, without knowing the results). Those that are replicated go into a higher acceptability tier. Those that do not get published in another tier. If replicating is not possible (for instance in the case of the LHC), the paper should go through 3 independent and simultaneous peer reviews.
The problem is not enough “gatekeeping”. Accepting a publication should be more stringent.
Okay, sounds reasonable. Except that peer review is a volunteer activity. Who is going to pay for all of these replication studies? Where does that money come from? If we can’t increase funding budgets, which are already paid extremely paltry, you’ve effectively cut research output in half. Maybe that’s a win, maybe not.
This reminds me of the welfare debate. On one hand, we could erect a giant bureaucracy to take a fine tooth comb over every application to make sure only the “worthy” get aid, which creates holes through which needy people fall. Or we can accept that driving fraud to 0 is impossible and get rid of the bureaucracy, then put the savings toward more aid. Sure some unworthy people will get it, but others who are worthy (and otherwise wouldn't have) will, and that’s more important.
If it’s standard practice that a publication involves replication, it becomes a standard part of funding. Will we have fewer studies? Maybe. Will they be much higher quality? Yes.
This is not like welfare. We are trying to enact a stringent gatekeeper so studies are completely reliable. The analogy doesn’t work here.
I've often said that grants should include funding for at least two independent reproductions. Also, journals should prioritize publication of negative results on par with positive results.
Trial pre-registration and open science are good first steps though.
The problem is not enough “gatekeeping”. Accepting a publication should be more stringent.