> Alpha: Statisticians use a "confidence interval" as a way to communicate how uncertain they are about a particular result. In our trial we might say "patients were 15% less likely to have the disease after taking the pill, give or take 2%". We don't think the decrease is exactly 15% (what we observed) but is instead somewhere in that neighborhood. Alpha is a measure of the chance the real effect is OUTSIDE of your confidence interval. So in this case, the chance the effect is < 13% or > 17%.
I know this sounds intuitive, but it is wrong.
The true effect is not a random variable.
The random variable is the statistic.
When we say "95% confidence interval", we are referring to the fact that 95% of the confidence intervals constructed based on the sampling distribution under the null will contain the true effect, not the chance that the true value is in the specific confidence interval you constructed.
Edit: The latter is either 0 or 1 but you don't get to find out in the context of a single test.
I know this sounds intuitive, but it is wrong.
The true effect is not a random variable.
The random variable is the statistic.
When we say "95% confidence interval", we are referring to the fact that 95% of the confidence intervals constructed based on the sampling distribution under the null will contain the true effect, not the chance that the true value is in the specific confidence interval you constructed.
Edit: The latter is either 0 or 1 but you don't get to find out in the context of a single test.