I recently read the book "Common Errors in Statistics." It was really illuminating. They spent a particularly long time discussing errors in various medical studies, although I think that's because the authors' backgrounds are in that area rather than medicine having particularly bad statistics.
To attempt to summarize the book, it seems that statistical errors tend to boil down to one of three problems:
* Improperly building the experiment, failing to ensure independence or account for confounding variables.
* Misunderstanding significance. Reading too much meaning into p-values, and performing post-hoc analysis. Remember that a 95 percent significance level means that in 5 percent of the trials, you'll get a positive result by random chance.
* Improperly using tests, by failing to account for their limitations. For instance, using a test that assumes normality when you don't have a strong case that the data is normal. Or using a test that requires a certain sample size on a dataset that is too small.
When you think about it, most scientists receive a few semesters of statistics strung out through their education. They learn a little about probability, then how to do a few significance tests and something here and there about regression. They then start doing work where in pretty much every field nowadays, publishing a paper involves statistical analysis at some point. And most of them grope their way through it based on hazy memories of introductory stats classes taken years before.
I don't like the title of this post, just because they're really pointing out that the correlations are actually not statistically significant, they're in error. But, beyond that, it's a very good article with some very important information contained within. We are approaching a point where neuroscience is getting good enough that we will finally be able to abandon the non-objective "social" sciences for actual science, and have a much better chance of approaching the truth.
I fear that the main point of the paper discussed might end up being largely ignored, however. This would certainly not be the first time where it has been shown that a certain level of 'accepted' evidence is simply not objectively true, but a large portion of the community continues to rely on it simply because finding the truth is difficult. The real problem comes when this sloppy research is used to drive public policy. Law enforcement is chomping at the bit to use fMRIs as lie detectors, for example. They have consistently ignored the scientific invalidity of polygraph examinations, of voice modulation tests, and I expect no better from them with regards to fMRI use.
It's also a fantastic real world example of "lying with statistics" that is subtle and easily overlooked. The simple presentation of a subtle (and possibly misguided) descriptive statistic can easily mislead a casual or biased reader.
To attempt to summarize the book, it seems that statistical errors tend to boil down to one of three problems:
* Improperly building the experiment, failing to ensure independence or account for confounding variables.
* Misunderstanding significance. Reading too much meaning into p-values, and performing post-hoc analysis. Remember that a 95 percent significance level means that in 5 percent of the trials, you'll get a positive result by random chance.
* Improperly using tests, by failing to account for their limitations. For instance, using a test that assumes normality when you don't have a strong case that the data is normal. Or using a test that requires a certain sample size on a dataset that is too small.
When you think about it, most scientists receive a few semesters of statistics strung out through their education. They learn a little about probability, then how to do a few significance tests and something here and there about regression. They then start doing work where in pretty much every field nowadays, publishing a paper involves statistical analysis at some point. And most of them grope their way through it based on hazy memories of introductory stats classes taken years before.