On: Psychology Journal Bans Significance Testing

[WhitneyLand]

Sorry maybe I'm being dense, but who would take 1 out of 20 success to mean they should start buying rabbit feet?

[Mvandenbergh → WhitneyLand]

Nobody. The problem is that if you conduct 20 trials of the efficacy of rabbit feet, you'd expect 1 trial to show a significant effect. (if you're interpreting P-values the way many people do, incorrectly.)

[DanBC → WhitneyLand]

When you do a single trial you might show that rabbits feet are effective.

You don't yet know if this is a 1 in 20 result or a 19 in 20 result.

That's why you replicate.

[ejain → WhitneyLand]

You'd start buying rabbit feet because only the study that was "successful" is published.