The 5% figure means that, when there is no signal to detect, we have a 5% chance of falsely claiming there is one. It does not say anything about the case when there is a signal and we do not detect it, which is known as the type II error rate.
With reasonable assumptions about sample size and the fraction of times there really is a signal, you can find that the majority of published results are false:
The 5% figure means that, when there is no signal to detect, we have a 5% chance of falsely claiming there is one. It does not say anything about the case when there is a signal and we do not detect it, which is known as the type II error rate.
With reasonable assumptions about sample size and the fraction of times there really is a signal, you can find that the majority of published results are false:
http://journals.plos.org/plosmedicine/article?id=10.1371/jou...
I have written an intuitive explanation of this, and a bunch more, in my book: https://www.statisticsdonewrong.com/p-value.html