IMO, the most prominent/important critique in thesis in this space was Karl Popper. He, for example, championed falsifiability for scientific theories.
Popper's two big targets for his criticisms were marxism and freudianism, who he accused of pseudo-science. Despite making a lot of enemies, his criticism made an impact and the fields evolved to address them, to some extent. They often dropped the "science" claim, or adopted more scientific methodologies. Today, Popper's pseudosciences are often characterized as (or evolved into) "soft sciences."^
Still, I think it's telling that 80 years later these general areas of study (economics, psychology, sociology, policy research) contain the majority of the problems this article is talking about.
A big part of the problem is statistics. That is, they are studyings tatistical phenomenon. The relationship between the "big hairy theories" which (eg supply & demand) and the more scientific/falsifiable hypothesis that they can actually test is.. problematic.
So... part of the problem is fixable... The scientific framework needs to be designed for statistics. Expiremental design plans can be published prior to expirementation, for example. Negative results need to be published too. Part of the problem is harder. Without a huge increase in independant replication studies, many of these fields are not going to be genuinely producing scientific knowledge, as fields. Individual results are more akin to anecdote.
The bigger problem is that "big" theories (eg keynsian macro, Maslow's hierarchy, etc.) are not generally scientific theories. That split between "small" testable theories and interesting, fundamental big theories is just very hard to bridge in a fundamental way.
^He went easy on liberal economists, possibly because of personal friendships... mostly because they didn't claim to be scientific to the same extent. I think in retrospect, this may have been a mistake.
Popper’s approach still works here. When some one writes a statistics based paper, they basically have said “I have a hypothesis that X correlates with Y. If this is not true (falsifiable), X will not correlate with Y in this data”. The hypothesis survives the test and this is science.
Of course, the authors probably did the opposite, but this doesn’t matter for the philosophical underpinnings.
So how do we detect bad studies? With this same framework. Take the same hypothesis, but design a different test. If the hypothesis fails the treat, publish that. That is science.
We shouldn’t be so hung up on reproducing old papers or scrutinizing every study. You’ll get lost in the details. If a hypothesis is true, it will stand up to every test of it. So if you doubt a result, test it!
Popper's two big targets for his criticisms were marxism and freudianism, who he accused of pseudo-science. Despite making a lot of enemies, his criticism made an impact and the fields evolved to address them, to some extent. They often dropped the "science" claim, or adopted more scientific methodologies. Today, Popper's pseudosciences are often characterized as (or evolved into) "soft sciences."^
Still, I think it's telling that 80 years later these general areas of study (economics, psychology, sociology, policy research) contain the majority of the problems this article is talking about.
A big part of the problem is statistics. That is, they are studyings tatistical phenomenon. The relationship between the "big hairy theories" which (eg supply & demand) and the more scientific/falsifiable hypothesis that they can actually test is.. problematic.
So... part of the problem is fixable... The scientific framework needs to be designed for statistics. Expiremental design plans can be published prior to expirementation, for example. Negative results need to be published too. Part of the problem is harder. Without a huge increase in independant replication studies, many of these fields are not going to be genuinely producing scientific knowledge, as fields. Individual results are more akin to anecdote.
The bigger problem is that "big" theories (eg keynsian macro, Maslow's hierarchy, etc.) are not generally scientific theories. That split between "small" testable theories and interesting, fundamental big theories is just very hard to bridge in a fundamental way.
^He went easy on liberal economists, possibly because of personal friendships... mostly because they didn't claim to be scientific to the same extent. I think in retrospect, this may have been a mistake.