Not quite. The 5% represents the chance that if the null hypothesis is true, you would draw data at least as extreme as the data you just saw in a repeated experiment.
Computing the probability that the data came from the theory stated in the null hypothesis would require a (Baysian) prior.
Also, Tloewald's reply is completely and inexorably wrong. Tloewald seems to want a Bayesian answer, which frequentist statistics can't give you.
"Given the evidence, there is a >=5% probability of the null hypothesis being true"
and
"There is a >=5% probability that if the null hypothesis were true, that your data would be at least as extreme"
The only difference I see is how you avoided saying anything about the null hypothesis, but I don't see how you can avoid saying anything about it.
if the h0 were true, then the probability of the result is unlikely, how can you not conclude that h0 is unlikely? What step are you missing other than collecting a preponderance of evidence against it?
The article never enters into this distinction. It makes it clear that people misinterpret evidence against the null hypothesis as evidence of the alternative, which is a false dichotomy.
I am confused. I also have sympathy for Tloewald at this point.
Sure. Let H0 be the null hypothesis and D be the data you observed. The first statement is P(H0|D) = 0.05. The second is P(D|H0) = 0.05.
The two quantities are related to each other via Bayes rule:
P(H0|D)=P(D|H0)P(H0)/P(D)
So indeed, as P(D|H0) goes down, so does P(H0|D). But if P(H0)/P(D) is sufficiently large, you can easily have P(H0|D) high while P(D|H0) is low.
I too have sympathy for everyone confused by frequentist stats - they tend to answer the exact opposite question that one really wants answered. In contrast, Bayesian stats tend to answer the question that most people ask.
P(D) is the probability of observing the data you just saw, due to either the null or non-null hypothesis. It's a strictly Bayesian quantity, since it's dependent on a prior. If your model has only a null and alternative hypothesis, then:
Since I just learned yesterday that statisticians precisely distinguish between "probability" and "confidence", kudos for using it correctly. At least I believe you used them correctly.
Computing the probability that the data came from the theory stated in the null hypothesis would require a (Baysian) prior.
Also, Tloewald's reply is completely and inexorably wrong. Tloewald seems to want a Bayesian answer, which frequentist statistics can't give you.