I worked with Nick a few times in a previous job and he was a reviewer on a paper I wrote; he was really nice and professional, and his feedback was constructive and incisive.
The movement he's part of is fundamentally disagreeable (in the 'big five' sense [0]), and some of its prominent characters, as you might expect, have reputations for being interpersonally difficult [1]. Collectively they've been labeled "data thugs" [2] and accused of "methodological terrorism" [3], which is why I think Nick's fundamental goodness is an especially valuable asset. We'd gain a ton of social value from certain reforms to the scientific research apparatus, and, IMO, being kind when delivering pointed critiques helps a difficult pill go down. That's different than pulling your punches; it just means staying focused on the thing you actually want to change.
> being kind when delivering pointed critiques helps a difficult pill go down
While I am sympathetic to this, I think it fails to recognize a very important thing that we expect, or at least should expect, from people who claim to be doing science: that every scientist is responsible for sanity checking their own work before making any claims based on it. As Richard Feynman said, your first duty as a scientist is to not fool yourself--and you are the easiest person to fool.
That means every scientist needs to be an expert in whatever fields are relevant to the work they're doing. If a scientist is going to make claims based on some purported correspondence between human psychology and fluid dynamics, they need to be an expert, not just in human psychology, but also in fluid dynamics; and if they're not, they need to not make the claims, no matter how enthusiastic they are about them. And the scientists who published the claims that Nick and his colleagues debunked were not experts in fluid dynamics, and knew it, yet they chose to publish anyway.
And that kind of thing, if science as an institution is going to be trustworthy, cannot be handled by kind words when delivering pointed critiques. It has to be labeled as what it is: not just being mistaken, but being scientifically dishonest, by making claims that you do not have the expertise to make. In a sane world, scientists who did that would be stripped of the label "scientist", the same way we disbar dishonest lawyers or revoke the medical licenses of dishonest doctors. At the very least, it justifies language that does not include kind words, but the opposite.
I have no problem with using kind words when calling out a scientist who is just mistaken. But I don't think kind words are called for when a scientist knowingly publishes claims that they don't have the expertise to evaluate for themselves.
> ... I don't think kind words are called for when a scientist knowingly publishes claims that they don't have the expertise to evaluate for themselves.
But, until you can establish beyond all reasonable doubt that that is what happened, you should be polite because there is a chance you are the one who is wrong. Then, after you have established it beyond reasonable doubt, you should be polite because trying to destroy someone will only make them dig in and fight you til the bitter end. And, because it's the right thing to do.
You can be kind and tenacious, and forceful, and not take no for an answer. I think that's what the original commenter was implying.
> until you can establish beyond all reasonable doubt that that is what happened
In the case under discussion, the key scientist involved (Frederickson) admitted that she wasn't an expert in fluid dynamics. That is sufficient to establish beyond reasonable doubt that yes, what I said happened is what happened.
> You can be kind and tenacious, and forceful, and not take no for an answer.
I'm not sure that "kind" is consistent with all of those other things, in the situation under discussion. But maybe we have different interpretations of what is "kind".
I thought I was going insane before reading your comment. This world is such a nightmare to live in.
Honestly, the idea that this guy even discovered something is just as ridiculous as the fact that the ruse had gone on for as long as it did. The only real story here is about the work that wasn’t done to begin with.
This is an interesting stance to take on HN, of all places. "Be kind. Don't be snarky. Have curious conversation; don't cross-examine. Please don't fulminate. Please don't sneer, including at the rest of the community."
This extends to other places, too.
On a more practical level, not being kind doesn't buy you anything. (I want to be clear here that "kindness" doesn't mean "freedom from consequences". Of course there should be consequences. But there's no need to be a jerk about administering them - it achieves nothing)
> This is an interesting stance to take on HN, of all places.
I am not talking about internet forum discussions in which the default assumption is that all participants are having the discussion in good faith. I am talking about what the response should be to a scientist who wilfully violates the norms that are required of all science if science is to be reliable and trustworthy.
The closest analogy in the context of an internet forum would be how a forum moderator should deal with a participant who wilfully violates the norms that are required to have a useful, good faith discussion. We normally call these people "trolls", and for ordinary participants the best thing to do is usually to ignore them, but a moderator has to maintain the forum's signal to noise ratio, which at some point is going to mean shutting the troll down, and doing it visibly and publicly, so that the norms of the forum can be seen to be enforced. Kindness would not be appropriate in that situation either (although since the situation is not as serious as a scientist wilfully violating the norms of science, one would not expect the response to be as vehement either).
> Of course there should be consequences. But there's no need to be a jerk about administering them
Publicly enforcing norms that are required for an institution to function, and making it explicit that that is what you are doing, in language that reflects the seriousness of the violation, is not "being a jerk". Granted, it's also not being kind. But "kind" and "jerk" are not the only available options.
Basic civility, respect and decorum go a lot further than mere "kindness" IME. The latter I can't even properly define in the context of a computer-mediated debate. I suppose it's largely a way of saying "don't personally attack other users, or you'll get booted" which ought to be plain common sense.
As I see it, being kind means being empathetic, asking ourselves how the other person is going to feel when receiving our communication, and therefore adjusting in a way to avoid emotional damage. I believe this is especially important when being critical. The goal of a criticism should be to help the other/the community/the discourse grow, keeping our ego out of the equation.
Agree. Expanding on ‘emotional damage’ - I’d say principally this would be softening the blow by not putting someone’s sense of who they are/sense of worth under threat.
Failing that, you get one of two counterproductive effects:
a) They feel a compulsive need to deny the threat and defend against it, doing something dumb or unwarranted as a result.
b) They lose that sense of worth, and become impotent. Recovery from this depends on their environment and ability to build themselves back up.
That said, if you’re too ‘nice’, there’s a chance you’re being too subtle and the message doesn’t get through.
Oh, I agree for sure that we shouldn't taunt or bait other users, in ways that would hurt them emotionally and tempt them to attack in turn. That's just as bad as an overt attack - it destroys the spirit of a robust debate. But this all falls under a proper understanding of respect and decorum, at least as far as I'm concerned. These words just feel more precise and accurate when referencing these things. Which helps making the norm stick.
All the more reason for the self reflection that went into that document ;-). I think these days, he pretty much follows it, at least in written communications. In person he used to get upset easily, which could lead to unkindness. I don't know if he is still like that, N years later.
Sorry, I didn't mean to imply that the author was a hypocrite.
I just meant, even if he was a hypocrite, that wouldn't necessarily undermine the text. A world class coach doesn't have to be a world class athlete. Or someone not practicing what they preach doesn't necessarily invalidate what they preach.
If this generalizes, seems like someone with a graph database, some basic calculus, and an archive of papers citation data could pull the rug out from under entire disciplines, specifically ones that are informing public policy these days. These "data thugs," they seem like what we hoped hackers would be.
For example one of the "data thugs" (James Heathers) wrote a tool that could detect impossible numbers in psych papers, like means/std devs that couldn't possibly have been the result of any allowable combination of results. Some very high percentage of papers failed, I think it was 40%.
And of course psychology isn't the worst. Epidemiology is worse than psychology. The methodological problems there are terrifying. Good luck getting public policy to even accept that it's happened, let alone do anything about it.
"Specifically, the mean of the 28 participants in the experimental condition, reported as 5.19, cannot be correct. Since all responses were integers between 1 and 7, the total of the response scores across all participants must also be an integer in the range 28–196. The two integers that give a result closest to the reported mean of 5.19 (which will typically have been subjected to rounding) are 145 and 146. However, 145 divided by 28 is 85714217.5, which conventional rounding returns as 5.18. Likewise, 146 divided by 28 is 42857121.5, which rounds to 5.21. That is, there is no combination of responses to the question that can give a mean of 5.19 when correctly rounded."
(that example is a fictional one but the same issue arises elsewhere)
No. Virtually no scientific papers with errors are ever retracted, in any field. That's how you can end up with fields where more than half of all claims are probably false. We're told science is "self correcting" but that's just one more lie on top of so many others. In reality science doesn't get corrected even when people go above and beyond to try and correct it, as in this story.
And people wonder why faith is lost in science. I wonder if we're at the point yet where anecdotal experience is more likely on average to be correct than a study.
Ironically, one of the areas of psychology the field has struggled the most to accept is also one with the most robust results and largest effect sizes. That area is ... stereotype accuracy.
It's exactly what it sounds like and it shows that, surprise, anecdotal stereotypes people have about other people are actually pretty accurate when tested. This is not a politically correct conclusion so the hard-left academic world struggled for a long time to accept this (and arguably still does).
"This chapter discusses stereotype accuracy as one of the largest and most replicable effects in all of social psychology. This chapter is divided into three major sections. The first, History of Obstacles to Social Psychology Accepting Its Own Data on Stereotype Accuracy, reviews some of the obstacles social psychology has faced with respect to accepting that stereotype (in)accuracy is an empirical question, and that the empirical data do not justify assumptions, definitions, or declarations that stereotypes are inaccurate. The second, The Empirical Assessment of Stereotype (In)Accuracy, summarizes what is now an impressive body of literature assessing the (in)accuracy of racial, gender, age, national, ethnic, political, and other stereotypes. The third, Stereotype (In)Accuracy: Knowns, Unknowns, and Emerging Controversies, summarizes broad and emerging patterns in that body of literature, highlighting unresolved controversies, and identifying important directions for future research. (PsycInfo Database Record (c) 2020 APA, all rights reserved)"
Not that simple, and that was something missed in the original article - not all citations are equal. You have to know the context of each. Like, it could've been 1 positive citation, and 349 calling it bullshit... or vice versa.
Indeed, the beauty of using a graph is you can see the direct citations, and then secondary and teritiery ones and then pull apart the clusters to see if there is any opportunity to pull on mathematical threads.
A query like, given paper X, show papers that cite it, papers that cite those, and then ones that cite both. Cycles in the graph might show citation rings as well (as I think has happened in the past), but this would be a challenge for young people to apply their high school calculus skills to verifying claims in social science research.
A breadth first search across papers using citations selects which paths you want to do your depth first validation on. We do the same thing in vulnerability research with software dependencies, and picking a bad-math vuln in a discipline and tracing its impact though, or picking clusters of tightly dependent research and finding vulns in them seems like an opportunity for an enterprising outsider who never wants a job from any of these people.
Scientists almost never negatively cite papers, and actually never do for incompetence or bad methodologies, so this is not a problem in practice [1]. In fact they don't even notice when citing retracted papers [2] so good luck getting them to notice deeper problems.
The reality is, a citation is always either neutral or a positive reference to other work.
Good points, and good to see there's some data behind that as well. That's how it's seemed to me in my areas. I'm not sure to what degree it applies for things like psychology - prob never as extreme as my original example, but it would be interesting to see a similar study.
Hahaha, it is a bit of a catch-22 though isn't it. In debunking a paper like this you have to understand that your are cutting away the foundation that people built their careers on, and you may be fundamentally damaging their academic trajectory. Doesn't mean you shouldn't do it, just that being a bit of an asshole probably helps.
Hopefully this will make future scientists more wary of building their careers on a foundation of shaky work that could disappear - and thus engage more critically.
There's a time for kindness, and then there's a time for brutal honesty. The Nick Brown story does usually feature another person who was too kind and ended up regretting it:
In his email, Guastello included a list of errors he had found in Fredrickson and Losada’s application of the math. “Ironically,” he wrote, “I did send American Psychologist a comment on some of the foregoing points, which they chose not to publish because ‘there wasn’t enough interest in the article.’ In retrospect, however, I see how I could have been more clearly negative and less supportive of any positive features of the original article.”
So. The people in the end who brought the problem to light were people like Sokal, who freely use terms like "bullshit" to describe what Losada and Fredrickson did. And the people who tried to be polite and kind and balance out their criticism with the positives, were blown off and got nowhere.
I'm unsure what your second citation is for, they don't use the word "data thugs" but the author certainly seems annoying (although likely correct about the points they're making as far as I read).
e: the parent comment has now been edited for clarity, keeping my comment for posterity
It is hard for me to comprehend a point of view where that article is not considered abrasive in tone. Whether or not it bothers you is another question.
The tile is in all caps "THE DUMBEST FUCKING THING I’VE EVER READ" and challenges someone to a cage fight at the end. The entire rest of the article could have been honeyed sweetness (which clearly it's not), and that could be enough to call it abrasive.
One person I showed it responded very favorably, commenting that the author clearly considered it "...important to get the basic points across violently vecause no one is listening to the non-violent arguments." i.e. this is abrasive, but abrasive to a purpose.
> If that's considered abrasive, we've really lost out on actual communication for the sake of bullshit etiquette.
Did we read the same article? I'm confident that we can have actual communication that is both less obnoxious and not hamstrung by "bullshit etiquette."
> its prominent characters, as you might expect, have reputations for being interpersonally difficult [1]. Collectively they've been labeled "data thugs" [2] and accused of "methodological terrorism" [3]
Ahh the "everyone I don't agree with/like are nazis" trope.
> the whole field of Big Five is fundamentally a fraud
Do you have more reading? The Big 5 is hardly perfect and merits critique, but to call it "fundamentally a fraud" frankly feels like the conclusion of a "topic tourist" that read 1-2 critiques and made up their mind. It's not theoretically-derived, it relies on factor analysis, its utility does not all cultures, the five traits are not fully orthogonal. I'm not here to champion it. But comparing Big 5 to Positivity Ratios or Power Posing seems hyperbolic at best, particularly given its usage in situations where validity must be demonstrated.
But I've changed my mind on a lot of things. I'm prepared to change my mind on this if you've seen or read something I haven't.
It’s been a while since I was studying this topic but if I remember correctly, there is fundamental problem of a lack of justification for operational definition in the whole field of psychometrics. This applies to most (all) personality tests including the big 5.
Because the whole theory is based around cluster analysis it is pretty important that you can justify the data that is going into the analysis if you want it to be reveling any truth outside of the model, otherwise you end up with what is called “junk-in-junk-out”. As far as I know, this justification is still lacking 40 years after this theory surfaced.
I think that the big 5—and personality psychology in general—might not have the same glaring issues as positive psychology. But solid science it is not. Fraud might be overreaching, but I would definitely categorize it as pseudoscience.
That's pretty subjective and could be leveled against almost anything behavioral or psychological.
The thing about the Big Five is that it has surfaced in all sorts of contexts. You can maybe say maybe the 5 per se is not well justified (as opposed to 4 or 6 or 7, for example), but if you take enough ratings of a person, some variant of those 5 will probably work as reasonable summaries of the ratings, and they will account for a substantial chunk of their predictive variance. The thing too, is that if you take other types of variables, like clinical symptom ratings, or diagnoses, you start to see roughly similar types of attributes become prominent.
The Big Five is a descriptive model of how people perceive others. There's a lot of evidence for certain mechanistic processes being heavily involved in some (e.g., positive emotion in extraversion, negative emotion in neuroticism, behavioral control with conscientiousness, etc.) but I'm not sure the original idea behind the Big Five was mechanistic -- it was a hypothesis about major dimensions that could summarize social perceptual data. It's like classification in biology pre-DNA era. People have some ideas of how things go together, and find it useful for organizing descriptions and measuremnts.
It's like if you did unsupervised DL modeling of all the videos involving humans you can find on the web, and found that their classification could be accounted for by 5 major vectors, almost all the time, regardless of sampling. Wouldn't you want to know that?
Many other measures are very mechanistically well-justified but lack ecological validity, in the sense that they are very narrow predictively and not well outside of laboratory contexts. That's fine, there's a tension there between predictive bandwidth and depth, but if you want any kind of rating of a human being's behavior and experience, you enter at your own risk if you think you'll measure something that's radically different from the Big Five (or something subsumed by cognitive measures). Can you do it? Sure, but a lot of the time no (see: grit).
What constitutes as an adequate justification for use of an operational definition within a model is subjective indeed. However there is usually a point in the life of a theory where the gathered evidence are sufficient enough that a scientific consensus starts to form that the operational definition is justified. I’m not aware that that has happened in the 40-odd year history of the Big 5 personality theory.
The 5 personality traits may be overarching within the field of psychometrics and they may indeed be useful to describe behavior, however you still need to justify that said behavior is not easier described using different models, and this is where personality psychologists usually fails in justifying their operands.
Works criticizing the model range from using totally different constructs (such as priming, positive reinforcement, universal grammar, brain dopamine level socio-economic status etc.)—which don’t rely on psychometrics at all—to claiming that the behavior psychometricians are predicting are actually not that useful (e.g. predicting ‘high confidence’ is not that useful if ‘high confidence’ does not result in a significant behavior which isn’t better predicted without made up operands).
If you were an early astronomer and you constructed the notion of ‘epicycles’ to simplify your model of planetary motion. You may use these ‘epicycles’ to justify your prediction, however you may not use a successful prediction to justify the existence of epicycles. Your epicycles may be useful until someone comes along and deems them unnecessary since planetary motion is better described by using elliptical orbits.
Of course this could go the other way as is the case with particle physics and the atom. However given the amount of research, success of rival theories, the failure of psychometricians from making useful predictions outside of their narrow field that isn’t better explained with alternative theories, I have high doubts that the Big-5 personality traits (and any theory of personality using psychometrics for that matter) is anything but pseudoscience.
> If you were an early astronomer and you constructed the notion of ‘epicycles’ to simplify your model of planetary motion. You may use these ‘epicycles’ to justify your prediction, however you may not use a successful prediction to justify the existence of epicycles. Your epicycles may be useful until someone comes along and deems them unnecessary since planetary motion is better described by using elliptical orbits.
I couldn't comprehend the discussion until I read this metaphor. Thanks for the detailed explanation.
I appreciate your response, but think I'd have to read more to be nearly as convinced as you. I'm familiar with the critiques you mention, but your conclusion as to the model's merits go beyond what I've seen other critiques make. As a model grounded in theory it is lacking, but as an explanatory construct to predict patterns of behavior and outcomes in specific settings (e.g., knowledge work) it persists for a reason. I've peer reviewed papers criticizing it, but none "debunking" it.
Personally my biggest gripe personally is that it is represented as a model of total personality, but that's definitely not true. It's just just representing a larger "personality space" than most other constructs. At least the outcome isn't placing one into a discrete category.
When I was doing psychology over a decade ago personality psychology was actually my biggest gripe. The way that I saw it, was that it wasn’t explaining anything which didn’t have better explanation using a different theory.
Now a decade later—being a little more class conscious—I can actually see how this is problematic. When a better explanation can come from sociology and has to do with class and economic status, measuring people based on a theory that lacks justification to justify one hire over another is problematic. When you ascribe it was because of an operationally defined concept (read: made up; based on data analysis) and name it personality that seems like a lame excuse to hire from your ingroup and excluding your outgroup.
These tests are robust, I know that, however robustness alone is not enough to justify a theory. These tools might be useful (or they may be dangerous) but while there is no justification for the operationally defined terms it cannot be used to justify terms outside of the model. That is pseudoscience. You can only to explain things inside of the model, and as such it is pretty limited as a theory.
I agree that the label "personality" unreasonably implies some sort of real (possibly for many, "genetic") truth.
> When a better explanation can come from sociology
Explanation for what? Behavior generally? In that case, the Big 5 is less of the predictor and more the criterion. The Big 5 is not a theory, it's a taxonomy that emerged from trait theory. It describes, it clusters, it predicts, but in and of itself it doesn't explain.
> ... seems like a lame excuse to hire from your ingroup and excluding your outgroup.
When it comes to hiring in the United States this is literally the opposite of the biggest use-case for personality testing for the past 50 years. But that's the only domain I can speak confidently on, and that doesn't include class or SES as a subgroup.
> there is no justification for the operationally defined terms
This is subjective, and I do not agree. If your critique of The Big 5 is actually a critique of trait theory more generally, I'm there for that. But a taxonomy for observable behaviors that shows reliability as well as content, convergent, and predictive validity across many populations and contexts seems justified to me barring a superior option.
The Big 5 is not and never would be a grand theory of human behavior, but it does describe actual behavior in a way that is interpretable and connected to the world we actually inhabit.
I don’t know how people interpret results from personality tests which gives them a criterion for what constitutes a good hire. However I have a strong feeling there is no sound science behind whichever criteria recruiters are using. And that risks placing arbitrarily high weights on whichever traits correlate with your in-group. This, however, is a falsifiable claim, and if anyone has ever done research which shows that personality tests actually reduces bias (as opposed to enhances it) then I’m open to be proven wrong. I do also question the efficacy of using personality tests as a tool to scope for good hires. How do companies actually measure that? There are so many biases that comes to mind that could make a company overvalue the efficacy of these tests.
My critique applies for all theories of personality. I don’t see personality traits as a useful categorization to predict behavior. By far the most research I see in personality psychology is about correlation with other terms inside a very narrow scope (this also applies other sub-fields of psychometrics; including positive psychology). The behavior that personality tests predicts does not further our understanding of the human mind. A theory of behavior that fails to do that is a poor theory at best.
But I want to go further, I don’t claim that personality psychology is just poor science but pseudo-science.
>> there is no justification for the operationally defined terms
The personality traits in the big-5 are operationally defined. That is they are defined in terms of what the tools are designed to detect. This is useful inside your models (as evidence by the success of big-5) but this does not tell us anything outside of our model. Now if you go to the real world and find evidence that these terms exist outside of your model, then I would change my mind. That would be a pretty good grounds for a theory which describes something that your model predicts accurately. If you don’t then you have at most a useful construct that you can use in other theories (think atom before they proved it’s existence). IF you can’t even use your constructs outside of your scope, then there is not much value in it outside of a narrow scope and you are most likely doing pseudo-science.
My critique actually extends to all personality traits. Personality traits (if they exist) can at best describe a proportion of the variability within a narrow scope. Outside of that narrow scope it is actually just better to describe a person as calm and courteous as opposed to speculate where they stand in the agreeableness axis in the Big-5 personality trait. And this is what I mean when I say “lack of justification”. ‘Agreeableness’ has been operationally defined within a certain model. When you use it elsewhere you need to justify doing so. And there should probably be a scientific consensus about whether the justification is good enough. If not you are most likely doing pseudo-science
Note that "operationalization" implies a fairly specific set of epistimolgical and ontological approaches which do not necessarily require that what is being measured has a one-to-one correspondence to a 'real' entity.
Indeed. You can operationally define anything you want within your model. If done carefully, a good operational definition may simplify your model quite a bit. (A bad operational definition, on the other hand, will almost certainly make your model overly complex and can be quite detrimental).
When you use your model to infer about a real world phenomena you have to be careful how you treat your operational definition. If you use it to make prediction, you cannot make a claim that what your operand caused it, not until you go into the real world and find it. If your model is successful you may use your operand to describe your prediction, but you have to justify why your operand is necessary, a better model may exist which doesn’t use an operand at all.
A successful model is neither a sufficient nor necessary condition for proving an operand exists.
Consider the whole type A personality thing - it's pure nonsense from a scientific perspective (created by tobacco companies to explain why certain people [smokers] had more heart attacks.)
But at this point it's still a useful cultural shorthand to describe certain characteristics we subjectively experience in others.
Yes, but that's a) not the Big 5, and 2) the worst example of personality tests/models where you're put into a category (which will show very low reliability over time, even if your responses show some).
"Empirically, men and women to tend to differ on the trait that personality psychology calls agreeableness. More women show up as high in agreeableness than men.
[As an aside, I once wrote Nassim Taleb and the Disagreeables.
Nassim Taleb’s latest book heaps praise on the trait that personality psychologists call low agreeableness. . . I am pretty far out on the disagreeable end of the spectrum myself, but Taleb makes me look like a goody two-shoes.
Taleb came across the essay and tweeted this response:
There is this BS in this "disagreebleness" scale used by psychologists, unconditional of domain. Like most psych categorizations, BS. Many are socially gentle but intellectually rigorous & no-nonsense: others nasty in person but appear gentle in public . BS!
Can you share more or point to further reading? I originally came across the Big Five because a psychologist was using it to debunk the idea of "grit" as being a predictor of success. (The claim was grit was not a new concept and essentially just a repackaging of the conscientiousness Big Five trait.)
It would be interesting (and another example of irony) if the Big Five itself was debunked.
I go into many reasons why big-5 (and personality psychology in general) might be a pseudoscience in a nibling comment. Here I would like to add that what you might have seen big-5 proponents do to grit (a trait from positive psychology), other fields of psychology (such as cognitive psychology, neuroscience, social psychology, behavioral psychology, etc.) does to personality psychology (and psychometrics in general).
I’m not aware of any debunking claims in particular, but skepticism is plenty, and the usual accusations range from pseudoscience to poor science.
In my opinion the worst critics go into an alternative theory of personality traits, e.g. 7 axis instead of 5 or neurotism is actually something else etc. These are attempts to debunk big-5 and in my opinion they always fall short. It doesn’t take long for proponents of big-5 to debunk these critics as the statistics behind the theory of big 5 is solid. Personality tests are robust and they consistently yield 5 disjoint axis in cluster analysis.
A better criticism goes into the foundation of the theory, attacks the fact that personality (and the personality traits) are operationally defined with little evidence they exist outside of the models. The behavior that personality tests claim to predict often have better correlation to non-personality traits, such as your brain dopamine levels, class, education or economic status, anxiety or depression, prior stimuli and response etc. Behavior is better predicted by looking at the person not how the person responds on a piece of paper. Their tests might be robust, but their science is lacking.
I’m not a believer in personality traits in general, neither “grit” nor “agreeableness”. To me the findings of any psychometrician is unremarkable at best and racist at worst (See Stephen J. Gould (1996) “The Mismeasure of Man”). In a given situation how you respond is never dictated by your personality traits. If it explains any variability behavior the effect size is hardly ever significant, and if it is there are probably other factors—not measurable with questionnaire—will explain it better.
Great post and thank you for going in-depth. What types of behavior were you thinking of when you typed "The behavior that personality tests claim to predict often have better correlation to non-personality traits, such as your brain dopamine levels, class, education or economic status..."? I'm just curious about what has been studied.
It’s been awhile since I was doing psychology, but following is a list of the top of my head (note that I’m likely to misremember or misrepresent some of these):
* One good study demonstrated that a common drug used to treat Parkinson deices has an unfortunate side-effect of limiting dopamine levels in your frontal cortex. People who used that drug are way more likely to engage in anti-social behavior after they start treatment then before. It seems like dopamine levels in your pre-frontal cortex can severely affect your personality.
* Social scientists often create amazing studies where they are easily able to manipulate the environment in such a way that they get participants to e.g. lie, cheat, etc. Off course there is variability within these studies and you can argue that personality trades explains this variability, however I don’t know if that has been done, and if it has, I wouldn’t be surprised if you would find that other factors such as religiousness, socio-economic status, gender, age etc. explains a bigger part of the variability in those social science studies.
* If you are looking for a better general theory of behavior then cognitive science has a number of constructs which don’t have this problem of operational definition. For example research has shown that you can prime people with a stimuli to increase the chances the participants will respond to similar stimuli in a short timespan afterwards.
* Behavioralism is more like a philosophy then science and posits that you can explain behavior by looking at the reinforcement history of an individual. That theory has been somewhat debunked by cognitive science, however behavioralists are still doing some impressive research which shows how you can modify behavior by offering some reinforcement contingencies. The same variability applies though as with social the social science studies.
I tried to sway away from specific studies because it has been a minute since I was studying psychology. But I hope this list is still relevant and accurate, and that it inspires you to go look for the individual studies which backs these claims.
Do note that what I’m doing here is a bit unfair. I’m making grand claims backed by evidence I think exists, but pushing the responsibility of looking for these evidence onto the reader. This is not how science communication is supposed to work, and my only excuse is that I’m lazy.
I think this is a fantastic example of how a) anyone can call bullshit on science if it doesn't make sense and b) there are more and less correct ways of calling bullshit on established science.
You don't need to hop on a podcast or radio show in order to question the orthodoxy of a scientific claim, and in fact you shouldn't if you want to be taken seriously. Nick Brown would have hurt his case had he done that, and instead found a path that actually knocked down a faulty concept in a way that has lasting effects.
This is the process working. This is all okay, and welcome, IMO, though possibly not by everyone involved.
I'm not sure I agree. In his own words, he had to ship around for a respected authority so that it would be taken seriously.
Once he had that, the journal have a bullshit response that it was passed two months since the original article was published, and so not eligible for publication. Sokal through his connections, went to the CEO to get that decision over ruled (edit: by threatening that the journal would be humiliated by refusing to publish it). The original debunker would never have got that far on his own.
To me, this sounds like a process that is far from some sort of egalitarian "judge ideas on their merit rather than the person presenting them." If he hasn't got Sokal involved, I wonder if this would have gotten published at all.
Edit: also, this casts a pretty negative light on the field, given that it's been cited 350+ times in this field. Not one of those other papers dug into the content (or understood it) enough to question it, and/or raise the alarm bells. It's one thing Its one thing if this was some obscure paper that never got eyes on it, but that is obviously not the case.
I agree completely with criticisms of the journal - clearly their process is flawed in a multitude of ways. But regarding "shopping around for a respected authority to be taken seriously" - I think this is actually an important lesson.
The problem with the vision of an egalitarian meritocracy in science (or any other field, honestly) is that reviewing, understanding and critiquing someone else's work has a non-zero cost - actually quite a large cost if done correctly. Also, the people most qualified to do this analysis generally loathe doing it, as they'd rather be spending time on their own original work. Inevitably, people develop heuristics to estimate the credibility of every new email or article that comes across their desk, to decide which is even worth reading.
Understandably, one of those heuristics is credentials/authority - the ratio of good articles published by non-experts in their field is extremely small and massively outweighed by crackpots. So what does a smart non-expert with a good idea do to hack this system? "Bootstrap" your authority, by first finding someone with more credentials than yourself, but not so much notoriety that they're too busy to read your email. If you can convince them, they'll bring it up the chain to the next-highest authority they can convince, and eventually it'll have some important names behind it and people will pay attention. Brown was able to accomplish this with only two collaborators, so I consider him a master bootstrapper for this :)
Everyone wants to be the patent clerk slaving away in isolation who pops out with a new revolutionary theory of the universe, but it's just not realistic these days. On the internet, a non-expert trying to rapidly gain credibility on their own is indistinguishable from social media hucksterism/"influencing". If you want to break into a new field, find some experts in that field and work with them!
"Everyone wants to be the patent clerk slaving away in isolation who pops out with a new revolutionary theory of the universe, but it's just not realistic these days."
And note that that patent clerk wasn't just some patent clerk. He had published four papers (plus a bunch of reviews) and finished his dissertation by 1905.
Juxtaposing this to Nakamoto's creation of bitcoin and original use cryptography, it makes no sense to me how anyone would have ever paid even an ounce of attention to what some random, supposedly unkown guy on the internet has to say about ... anything.
Or how Perelman even got someone to even eant to verify his solution. Maybe not so much Perelman since he had some important figures knowing him already.
Bitcoin received early support from high-reputation folks in the cryptography community, including Hal Finney and Nick Szabo. (As a result, some people thought that one or both of them were Nakamoto.)
Regardless of who Nakamoto actually was, it does appear that Finney and Szabo played a similar role that Sokal did in this story; getting support from high-rep folks was essential to getting Bitcoin off the ground.
Perelman was widely known as a math prodigy (he won gold for the USSR at the International Mathematical Olympiad) and studied at the Leningrad University, so he could approach pretty much any of his former professors to take a look. He was definitely not an outsider in the field. A recluse, yes, but that is different from being a no-name.
Yeah Bitcoin is such an interesting exception! I think some ideas are viral enough on their own to go from zero to front-page on HN/reddit/etc. even when posted by an anonymous user - but it's rare to go beyond that, they usually run out of momentum.
Bitcoin had a few things going for it - the whitepaper is concise and well-explained, in perfect academic prose and typeset in LaTEX, which lends it an air of credibility. It came with an implementation, so hackers could experiment with it immediately, plus it sort of has implicit libertarian anti-government undertones, both of which encouraged follow-on blog posts.
Once the ball got rolling, another kind of heuristic took over - the money heuristic. People put money into the thing, which made other people go "oh look at how much money they put in, this is a real serious investment thing!" and put in their money, in a sick feedback loop that continues to this day...
> Not one of those other papers dug into the content (or understood it) enough to question it, and/or raise the alarm bells.
Quite the opposite, the article itself found out that many of them did question it (and quotes three such researchers), however, "raising the alarm bells" and disputing such claims requires exceptional diligence (more than the original article did) and lots of thankless work (like Nick Brown and his two collaborators did) that's not likely to be rewarded (you'd be lucky if you can even get it published in an appropriate venue); so they quite reasonably went on with their own research agendas instead of letting their actual work languish while doing a debunking campaign.
Science can only be as good as the people doing it.
The problem is that most papers, especially on very complicated topics, are not science and just serve the purpose of proving a point or helping someone's career.
Even without taking easy jabs at psychology (you would need full brain understanding of its inner workings and evolution over a lifetime, to have a 100% clear picture of what's happening - we probably have 0.0001% - it's all based on observations and interpretations), the reproducibility crisis is affecting a lot of areas.
Hopefully with more time and more people getting involved in the field, we'll get more and more science.
If I think about nutrition / fitness, between 40-20 years ago and now we made great progress: we went from "Fats are bad and make you fat" (pure propaganda to sell processed sugars), "BULK on a 4000 diet" to any video of Jeff Nippard quoting incredibly detailed studies about minutiae of nutrition and building muscle.
The journal publishers (similar to FB et al) are commercial entities designed to make money from desperate academics trying to publish. They thrive on novelty and readership, but also "respect" (rather than ads and "engagement") from not having their occasionally outrageous papers questioned. Sometimes to stoke or avoid scandal and increase readership they might include a debunking paper, but it's important to remember these are not bought by individuals but by departments who won't stock "less respected" journals.
Now, if perhaps there were a respected public psychology paper service like psyarXive at the time that might have made things slightly better.
I'm not sure I agree. In his own words, he had to ship around for a respected authority so that it would be taken seriously.
I don't see how that contradicts the person you are replying to. Ze is saying there is a path to contradicting established science. Ze didn't say that it was easy, or that it didn't involve working with people who are in "the establishment" - just that jumping on podcasts and talk shows and spouting off as an outsider with no credibility isn't the best way.
If your goal is to try and reform from within then sure, spending months waiting for BS replies from journals is the best/only way.
If your goal is to spread the word that psychology is unreliable and you don't care much about what psychologists themselves think, then podcasts and talk shows are a far more effective strategy.
Psychology isn't really a very harmful field - bad claims in psychology mean maybe some people waste time on a useless self help method, or don't get proper psychiatry when maybe they should. But a lot of low level damage still adds up, and there's a lot of people who listen to psychologists. If you go the media (NB: which is what Brown is doing here), then you can potentially have a more positive impact on balance.
> This is the process working. This is all okay, and welcome
No! This sort of thing casts enormous doubt on the field of psychology and many soft sciences as a whole. A lot of it is academic politics and people being afraid of calling things out in fear of hurting their career prospects. It's much easier to just stay silent, ignore it, and play along. Who knows who will sit on your next grant proposal's committee. Academia is full of this and it's a disgrace.
I, for one, will never trust another “finding” coming out of the field of psychology.
Between this and the marshmallow study, I think psychology should be considered an art. We’re many centuries away from this progressing to being a science on par with physics.
There’s nothing wrong with that, a wise therapist has much to contribute - but not to science.
There's an unfortunate tendency to treat science as the only source of truth. Really, psychology (like engineering and medicine) is a profession. Professions aren't just based on the scientific method -- they draw from a semi-formal pile of experience known as "best practices." These are often just based on something that a clever person decided to try decades or centuries ago which seems to have not hurt anyone. Of course it is an iterative process so sometimes these things are overturned, but the process seems to work allright.
Spot on. I’ve met many psychologists who struck me as kind, wise people with a helpful perspective. Society needs them.
But to plug this fuzzy, professional knowledge into equations, call it science, and use it to make decisions in business and policy is at best naive and at worst fraud.
> psychology (like engineering and medicine) is a profession.
Throwing engineering in there with psychology and medicine is not a very good grouping of disciplines. At least not if "engineering" means the kind of engineering that professionals in the field have legal liability for (like the engineering that goes into designing bridges and buildings).
Engineering does have "best practices", but those practices have to operate within limits that are very well understood based on underlying physical science that has been thoroughly tested and for which we have mathematical models with excellent predictive capability. The "best practices" in psychology and medicine have nothing like that kind of support.
> the process seems to work allright
Engineering, at least the kind that designs bridges and buildings, seems to work all right, yes. I'm not sure I would say the same for psychology and medicine.
If you actually talk to engineers who have to follow regulation and building code, they will tell you how often the rules are nonsense and don't make technical sense, they have to click checkboxes to say that something is fulfilled that doesn't even make sense in the given context etc.
I certainly agree that many local regulations and building codes are not there for engineering reasons, they're there for political and economic reasons that have nothing to do with good engineering.
But it's also true that any of those engineers who say that a particular regulation doesn't make technical sense, will be able to explain to you in detail why it doesn't make technical sense, based on the kind of theoretical and practical knowledge I described, the kind that is backed by well-tested models with excellent predictive capacity.
I think the differentiation between "science" and "profession" is a bit crude. You obviously don't include mathematics in this, but it is a form of deductive reasoning as far away from science as the arts are. Deductive reasoning is just far less prone to fault than inductive reasoning, be it mathematics or philosophy, and the methods we have developed in some of the natural sciences over the centuries to model real-life behavior on mathematics are really great.
You are right that these largely don't exist in the Social sciences, but this is at least partially due to the much more complex subject matter. Still, both the social sciences as well as the natural sciences are trying to approximate reality through models built on experiments, and are thus fundamentally different from deductive reasoning in closed systems.
Not trying to mince words here, but engineering is applying validated knowledge to solve real world problems, sure, you can call it profession, but the fields generally associated with engineering are still in the business of generating knowledge. Medicine and Psychology heavily rely on the scientific method to validate abstract concepts, while engineering adjacent disciplines like Computer Science heavily really on maths and deductive reasoning to solve problems in the space of algorithms and computers.
You obviously don't include mathematics in this, but it is a form of deductive reasoning as far away from science as the arts are.
I disagree strongly with this. To reduce mathematics (and philosophy in your next sentence) to not-a-science, just because it's based on deductive reasoning is doing it a major disservice.
It is especially because of its deductiveness that mathematics is such a valuable tool for science. It answers the fundamental question "assuming A, what can we prove to be true about A' and A''?". Without that deductive proof, many inductive scientific theories could not even be formulated, let alone disproven.
And then you go on stating that engineering is applying validated knowledge to solve real world problems. Do you realize that much of that validation of said knowledge has been done through mathematical models? That the only reason we have certainty in engineering is because of the deductive rigour of mathematics?
Mathematics & philosophy aren’t sciences because there are no hypotheses, experiments, and results; it’s just thinking and logic.
It’s still incredibly valuable and the basis of many good things.
I would also add that engineering isn’t only working with existing scientific knowledge; we were building steam trains and optimizing them long before we started doing scientific thermodynamics.
Science and math are definitely good friends and so I'm sure lots of analogies between the two could be made, but I believe the comment was getting at the epistemological difference between 'proven' (the happy state in mathematics) and 'not contradicted by experimental evidence' (the happy state in science).
There was an interesting discussion on HN last week that brought up the fact that mathematics has its own crises, namely a communication crises. It was brought up that the proofs can be so dense that errors will be published and go unchecked for a long time. The interesting part is that everything that's needed for "replication" is literally within the paper and the errors still fall through the cracks.
There is still an awful lot of engineering that is model-based without strong deductive proofs. (Mechanical failure theory comes to mind). But the actual origin of "profession" comes from professing an oath to the public good. Meaning the traditional professions (law, medicine, engineering) aren't necessarily aimed at creating new knowledge, but applying knowledge to the betterment (or profit) of society. Sometimes that butts up against problems without known solutions that requires adding knowledge to the field, but that's not really the primary goal, unlike something like mathematics.
I'm really not sure what you are getting at. Psychology is pretty definitively a science. So is a massive chunk of medicine. Science holds extra weight because it involves at least attempting a verification step following an idea.
Treating science as gospel is stupid. But without personal experience (and frequently even with) stating something as true is largely meaningless. Stating something is true, while providing data to back it up, as well as evidence that other subject experts have checked that data may not be perfect, but that doesn't put it on equal footing with a gut check.
How could you possibly know that the "process seems to work alright" without scientific evidence that it's doing so? You actually have no idea in that case whether you're doing harm or helping, and if you're somehow helping, it could just be a very expensive placebo.
I mean, I don't work in mental health, but if somebody has mental health issues that they are dealing with, and they buy an "expensive placebo" in the form of talking about their problems with a professional, and it helps... that seems like a successful treatment, right? I'm actually not even sure how to define placebo in this context.
I think it's important to focus on "expensive" and not "placebo" in that statement. If a placebo is all that's needed (or all that we can do), then arguably it shouldn't be expensive.
If what you've read here concerns you to the point of throwing out the field, you're going to be immensely disappointed by every other scientific field, as well. [0] I'm not an expert, but my understanding is that this article could have been written about literally any field of science, with a depressingly few nouns changed. [1]
This is what people mean when they say, "science is messy". This is how the sausage gets made, and if you're cool throwing out psychology as a field, you're probably also going to have to be cool throwing out a lot of other fields. [2]
There are efforts being made to clean up how science is conducted, called "metascience", but it's a relatively new field and there's still a lot that hasn't been investigated yet. [3]
Basically, if you've ever thought of yourself as "pro science", you've been supporting what happened in this article, whether you realized it or not.
I think we can maybe inject a little bit of extra nuance here.
Not every field is the same, they vary massively in size, level of external influence, and complexity in subject matter. Psychology suffers more than most. It's a large field, so competition to get noticed is likely high. There is money to be made, as discussed in the article, so it struggles with influences outside of pure scientific inquiry, and they have no chance of describing their theories in terms of pure component mathematics.
Contrast that with ocean and atmospheric sciences. It is a tiny and very insular field, so the rat race works very differently. This also protects them from outside influences, and a lot of the major underlying physical and biological processes involved can be modeled with mathematics that have been demonstrated to be predictive.
It's just easier for some fields to reliably produce higher quality work. I think this is a case where blanket skepticism is just lazy. Every field suffers some level of a reproducibility issue, but some fields are better able to self select for a variety of reasons.
Thank you for saying this. I mean, Computer Science is up there in the junk science department. Coming from a phsyics background, it's hard to even call most of what happens in CS "science".
Go to any CS conference and you'll see a whole bunch of papers doing micro benchmarks, saying that their method is X% faster than all the others, where X% is usually some very low percentage (and of course no errors bars are visible on any plots). You talk with the guy presenting the research and you find out that half the benchmarks were done on one machine, while the other half were done on another with different specs. Well, okay, now why can I trust the benchmarks? And how many other papers are doing this? And is your result even meaningful anymore? It's so depressing.
And don't even get me started on the AI/ML field. My god, talk about art over science.
A lot of AI/ML work is sloppy but well known methods work reliably, textbook descriptions make sense. That there's a long tail of shitty student papers doesn't invalidate the field. Year after year methods genuinely get better. Whatever process produces this progress, it produces this progress.
> you're going to be immensely disappointed by every other scientific field, as well.
Not every other field. We do have scientific theories that are reliable. But the reason they are reliable is that they are nailed down by extensive testing in controlled experiments that have confirmed their predictions to many decimal places. (I am speaking, of course, of General Relativity and the Standard Model of particle physics.) So we do know what actual reliable science looks like.
The problem, of course, is that there is not a lot of other science that actually looks like that, but it still gets called "science" anyway, and treated as though it is reliable even though it's not.
I said "every" because even some of the best work science has done, General Relativity and the Standard Model, were to my understanding an incompatible set of theories.
I did a bit of brief Googling, and it looks like some of that has been resolved and it isn't seen as totally incompatible anymore, but even that's not locked down as of yet.
Of note, it looks like there's a debate ongoing even within the citations about compatibility/non-compatibility, which I've never seen before on a Wikipedia page (I've done some light editing to separate out the relevant quotes from their related citations to improve readability):
>> Sushkov, A. O.; Kim, W. J.; Dalvit, D. A. R.; Lamoreaux, S. K. (2011). "New Experimental Limits on Non-Newtonian Forces in the Micrometer Range". Physical Review Letters. 107 (17): 171101. arXiv:1108.2547. Bibcode:2011PhRvL.107q1101S. doi:10.1103/PhysRevLett.107.171101. PMID 22107498. S2CID 46596924.
> "It is remarkable that two of the greatest successes of 20th century physics, general relativity and the standard model, appear to be fundamentally incompatible."
>> But see also Donoghue, John F. (2012). "The effective field theory treatment of quantum gravity". AIP Conference Proceedings. 1473 (1): 73. arXiv:1209.3511. Bibcode:2012AIPC.1483...73D. doi:10.1063/1.4756964. S2CID 119238707.
> "One can find thousands of statements in the literature to the effect that "general relativity and quantum mechanics are incompatible". These are completely outdated and no longer relevant. Effective field theory shows that general relativity and quantum mechanics work together perfectly normally over a range of scales and curvatures, including those relevant for the world that we see around us. However, effective field theories are only valid over some range of scales. General relativity certainly does have problematic issues at extreme scales. There are important problems which the effective field theory does not solve because they are beyond its range of validity. However, this means that the issue of quantum gravity is not what we thought it to be. Rather than a fundamental incompatibility of quantum mechanics and gravity, we are in the more familiar situation of needing a more complete theory beyond the range of their combined applicability. The usual marriage of general relativity and quantum mechanics is fine at ordinary energies, but we now seek to uncover the modifications that must be present in more extreme conditions. This is the modern view of the problem of quantum gravity, and it represents progress over the outdated view of the past."
> I said "every" because even some of the best work science has done, General Relativity and the Standard Model, were to my understanding an incompatible set of theories.
If you insist on treating either one as though it were a fully fundamental theory, capable of explaining absolutely everything, then yes, they are not compatible.
But you don't have to treat either theory that way in order to make the claim I was making. Even if both of those theories end up being approximations to some more fundamental theory, it remains true that, within the domains in which they have been tested, they make accurate predictions to many decimal places and have been shown to be thoroughly reliable. Certainly in any situation in which you might have to bet your life on one of those theories, you are going to be well within the domain in which they have been thoroughly tested. And within those domains, there is no incompatibility between them.
Yes, it has been debunked. They got the causal arrow around the wrong way.
It turns out that kids who are in stable situations with trustworthy adults around them, tend to trust adults more when they say “wait 5 minutes and I’ll give you more”, and that kids in bad situations don’t trust adults. Turns out being in a bad situation as a kid does have a long term effect on your life.
Nothing to do with the ability of people delay gratification.
More often than not the constructs in the garbage studies are quantifiable, just useless. The bigger problem is the "what" of what is being measured---whether that thing is representative, useful, or important. Part of the reason bad research has the veneer of authenticity is that the numbers, graphs, and statistics dress up the nonsense.
The messier the system, the more rigorous the methodology should be. Pre-registering trials to fight p-hacking, and open data should be the norm. That's decidedly not what we've gotten from this field, and some prominent figures have even been fighting such measures and consider the reproducibility crisis to not be a big deal. That's the problem.
I won't argue with your mistrust of psychological 'findings'.
But: If psychological research is not conducted with scientific rigor, and we treat it as an art, then where does that leave the phenomenology of mental illness, study of social dynamics, etc? Mentally ill people still need to be treated. And historically they have been treated in very, let us say, 'artistic' ways. If we abandon the scientific aspects of psychology, mental illness will still need to be dealt with, but without any framework of rigor or understanding.
While landmark findings of psychology (such as the marshmallow experiment) can be questioned and debunked, a lot of what you'll learn when studying psychology as a science is how little we know about psychology. That's an important thing to learn! Does approach X work for people with debilitating anxiety? In all probability, we don't know, or it does work for some people and has the opposite effect on others. That's the day to day of psychological research. When something exciting and novel comes up (like the marshmallow thing, or the Stanford Prison Experiment), it represents an island of false certainty in a sea of uncertainty and suddenly all the laypeople know about it. Such understanding of our lack of understanding can form a bulwark against, at the very least, political, social and religious ideologies that make sweeping claims about behavior and dynamics.
It's important to quantify what we don't know as well as what we know. Otherwise we'll be taken over by people making extraordinary and sweeping claims based on no evidence, rather than people at least publishing their evidence to be possibly later debunked. In the linked article's example, if psychology were purely art, there would only be the authors' book, not a paper with actual math to debunk and a journal to interact with. I am probably overstating my point, because when a specious claim is backed up by scientific trappings people may be more likely to believe it, but I am increasingly less certain of that.
When it comes to critiques of 'soft sciences' I often come to this. Those fields often came into being because there's actual day to day issues that don't go away even if we have arguable theoretical foundations behind them. Absent said foundations, people still need to make sweeping decisions about economics and psychology that affect millions of lives. The scientific part arose from looking at the history of such interventions and wondering if things like exorcising people with visions actually worked any better than not exorcising them. For many of the people being exorcised, it was a 'landmark finding' that exorcism is not the most effective method of helping them with their issues.
> If we abandon the scientific aspects of psychology, mental illness will still need to be dealt with, but without any framework of rigor or understanding.
I think false rigor is worse than no rigor at all.
> In the linked article's example, if psychology were purely art, there would only be the authors' book, not a paper with actual math to debunk and a journal to interact with. I am probably overstating my point, because when a specious claim is backed up by scientific trappings people may be more likely to believe it, but I am increasingly less certain of that.
I think the primary reason why people are less likely to "trust" scientific results now than in the past is precisely because of some scientists generating "scientific trappings" with the appearance of rigor to justify a bullshit result. Most of the time, no one cares, and those scientists go on to their academic careers as usual. Of course, it's the follow-up careers, books, headlines that lead to the erosion of trust, not the fake math itself (that's just necessary to gain the approval of other scientists).
When that is attached to something that the average person can blatantly see is wrong, it erodes their confidence in the entire scientific process and establishment. They'll reject the entire concept of rigor before believing some of the things scientists claim to be true. How is a lay person supposed to separate the modeling that goes into e.g. climate change and "a 2.9 positivity ratio is the key to happiness"? To them they look pretty similar, a bunch of mathy looking garbage that tells them something that isn't intuitive.
Personally, I've been forced to sit through a lot of corporate management trainings that are full of citations of titillating psychology results that I know are bullshit. I just don't have the patience or motivation that Nick Brown has to properly debunk them.
I quite agree with most of what you are saying, and you are saying it quite well. My main thrust is that we shouldn't throw the baby out with the bathwater by saying, 'Some people aren't rigorous so let's abandon rigor' (I don't think you're saying that, it's a strawman). Even in the paper critiqued in the original post, there was some rigor (in the midst of clear blindness and hubris), sufficient for the team in the article to take it apart on its own merits.
There's real examples of cases where no rigor is applied at all, and some facsimile of rigor can improve things in a psychological context. For example, much ink has been spilled in Hacker News about traumatic interview practices adopted and popularized by the FANG companies, such as so-called 'brain teaser' questions. Included in these practices was the widespread notion that interviewees needed to be 'put on the spot', or 'think on their feet'. Interview practice has been studied by the field of organizational psychology for decades, and such practices were from day one counter to several findings (such as that putting the candidate off-balance or in a state of discomfort during the interview and having them successfully navigate that would somehow predict job performance). Eventually several large tech companies conducted internal studies and concluded that the practices had no bearing on job performance.
I too have seen BS titillating psychology results, but the antidote is almost always to review the literature. For example you might often hear "conscientiousness on a Big 5 personality test correlates with job performance". Yes it does, but review the literature: it is quite weak and will not do a good job of explaining individual variance.
Let's say I had heard about this magical ratio in the OP's article in a job context. My BS meter would have gone off, most certainly. Some actionable number that can explain a tremendously complex and poorly understood set of dynamics? Hmph! I would have reviewed the literature and found the paper in question. I would have seen the large number of citations. Ok, that gives it weight as an idea, but let's see if the idea has legs. Where are the meta studies showing multiple independent corroborations of the number in different contexts and performed by different researchers? Non existent. As someone who takes science and applies it, for me that puts it firmly in the area of 'very interesting finding, but not actionable'. Honestly I think that's probably what was quietly happening with that ratio even before the retractions. Such findings make for great media news, corporate power points, motivational speech meetings, annoying HR meetings, etc., but hopefully (I do say hopefully :) ) in almost every real world setting if someone told a senior manager, "Your team is going to be measured on this positivity ratio, because science!", that manager would make a stink. Of course, maybe not. I do believe an important skill that needs to be increasingly taught is the ability to parse and understand the landscape of papers on a subject.
I agree.
The real shining light here is that a determined amateur was able to get free, volunteer help from inside the community to weather the publication process.
Those who read the review / revision process in this article and were dismayed / convinced the process was flawed should realize this is precisely how review works in many fields. It takes a year to get a paper reviewed and revised, and a "25 page" (TFA) statement of revision is par. Not "all" will have the stomach for that, but once you are part of that process, it's normal.
You consider having to threaten the reputation of an academic journal by emailing the CEO directly before you even make it to the review for a well written, concise, critical paper "part of that process"?
Not even mentioning that it required the publicity and reputation of a well known figure (Sokal) to even open that door for them.
If that's the "shining light" of this industry - I sure as fuck don't want to see the dirty alleyway.
You could claim that Harris Friedman is an insider - although I don't find a nearly retired (at the time - now actually retired) college professor to be particularly "inside" the journal space, but he is certainly a member of the field.
The people who were insiders (Barbara Fredrickson and the reviewers at American Psychologist) are decidedly unhelpful and uninterested outside of throwing folks under the bus and equivocating around how so much fucking fraud/bullshit ended up in their papers.
The people that publish and promote bullshit science spend a lot of time on podcasts, so I don't see why the people falsifying their research shouldn't as well.
It took 3 people months of work to debunk a claim.
These resources just don't exist in reasonable numbers and we absolutely need short, piercing, direct call outs that quickly demonstrate that fuckery is going on.
Feelings are the worst thing that has happened to science.
that's an odd interpretation. this seems less like a system doing what it's supposed to do and correcting itself and more like an instance of that system failing to do what it's supposed to do and correcting itself.
The article describes events set into motion over a decade ago. People calling bullshit on things that happened over the past couple of years have had a much worse time of it.
Taking equations from physics out of context to bolster social science claims isn't unique to psychology, other social sciences like economics have done the same thing with little more justification. The rationale behind this approach is simple marketing, in that 'using more mathy stuff' will sell better to the general public than non-mathy hand-waving claims will.
However, the practioners of this approach don't seem to understand that physical equations are constructed by analysis of empirical experimental and observational data on one hand, and theoretical mathematical development on the other, within limited scopes of applicability. Copying equations from one area of science to another willy-nilly like the article describes is just ridiculous - would cosmologists studying general relatively just plug in quantum mechanical equations in the hope of getting a 'mathy explanation'?
Here's another example from 'mathematically sound neoclassical economics', which does the same with 19th century physical science equations:
> "The strategy the economists used was as simple as it was absurd—they substituted economic variables for physical ones. Utility (a measure of economic well-being) took the place of energy; the sum of utility and expenditure replaced potential and kinetic energy. A number of well-known mathematicians and physicists told the economists that there was absolutely no basis for making these substitutions. But the economists ignored such criticisms and proceeded to claim that they had transformed their field of study into a rigorously mathematical scientific discipline."
>Because neoclassical economics does not even acknowledge the costs of environmental problems and the limits to economic growth
Econophysics is really crank stuff but some people really like to shit on econ without knowing a tenth of the discourse in the field. How can someone not have heard of carbon taxes or balanced growth paths?
What's funnier is that you both are revealing that either you don't know the methodology of solving economics models, or don't know what the physics equations represent. Nature acts to minimize convex functions while economic agents move to maximize concave ones. Of course the solutions end up the same.
Don't think that our utility functions are correct or actual people act differently? Honestly only having profits to maximize is just a good first order approximation. That's why we have an entire field working on supposedly irrational behavior and backing up everything with data and experiments.
Now economics does have its huge problems, but things aren't as simple as this. But if you still see a huge problem, just get your correct model, back your results with data and enjoy getting the Nobel prize.
> But if you still see a huge problem, just get your correct model, back your results with data and enjoy getting the Nobel prize.
The "Nobel prize" for economics isn't a real Nobel prize. It was made up to make economists feel better.
From Wikipedia:
> He explained that "Nobel despised people who cared more about profits than society's well-being", saying that "There is nothing to indicate that he would have wanted such a prize", and that the association with the Nobel prizes is "a PR coup by economists to improve their reputation"
> Nobel despised people who cared more about profits than society's well-being
Good thing the prize isn't for Business Administration then. Also ironic, since it's economics which studies anti-trust and gives legitimacy to strike down such practices.
Seconded. Blindly implying that economists of all groups are blatantly appropriating math from other fields without understanding it seems a step too far to me. I'm sure it happens, but I've worked in conjunction with people writing computer models to study certain economic subsystems and virtually everybody involved had a serious academic background in mathematics.
There are niche heterodox subfields of economics that literally are just a dubious physics analogy (thermoeconomics), but the source's argument that marginal utility theory was copying von Hemholtz is the sort of extraordinary claim that requires extraordinary evidence, not zero citations, baffling confusion of economist Francis Edgeworth with literary writer Maria Edgeworth and a lot of tacked on claims that are laughably false (Jevons certainly didn't believe that "natural resources are inexhaustible: he was actually initially famous for trying to predict when the coal supply would run out!).
There are obvious reasons for studies of how money is spent to involve mathematics (though Menger managed to deduce marginal utility theory from first principles without equations) and it's unsurprising some of the equation forms superficially resemble some of the many simple equations of physics. There are much better criticisms of neoclassical economics and the application of neoclassical economics than accusing the early economists of plagiarising another field!
Based on other people in the field whose responses were quoted at the end of the article, I suspect this phenomenon may still be occurring. Specifically:
"David Pincus, a psychologist at Chapman University who specializes in the application of chaos theory to psychology"
"Stephen Guastello, a psychologist specializing in nonlinear dynamics at Marquette University"
> In a valid fluid dynamics problem, the numbers plugged into the equation must correlate to the properties of the fluid being studied. But in attempting to draw an equivalence between the physical flow of liquids and the emotional “flow” of human beings, Losada had simply lifted the numbers that Lorenz used in 1963 to explain his method in the abstract, numbers used merely for illustrative purposes. Losada’s results, along with the pretty butterfly graphs Brown had been shown in class, were essentially meaningless.
Incredible. So it wasn't just a matter of, say, applying an equation beyond its domain, or generating the inputs to the function in a sketchy way, or hacking the inputs to the function until they got an answer they liked, but literally copying example parameters from a 1963 paper.
The "Complex Dynamics of Wishful Thinking" paper is very readable:
It is very readable, and it utterly nihilates Losada's papers. It is hard to believe that anyone could have taken them seriously, let alone let them get published. Sigh. Nice touch that Sokal helped debunking it.
I wonder to which extent this taints the whole of "Positive Psychology", which I was moderately excited about.
'I think the educational and psychological studies I mentioned are examples of what I would like to call Cargo Cult Science. In the South Seas there is a Cargo Cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. So I call these things Cargo Cult Science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.' - Richard Feynman, 'Cargo Cult Science'
I think there is inverse relation of how frequently a number is written everywhere with it's degree of decimal places. It is a convenience thing. Also if the number is generated on a computer and used there subsequently 4 decimal is plenty for social science and business research/purpose IMO.
Physicists got to that point (very roughly) by constructing complex mathematical models and building large machines to test those models.
In contrast, leading psychology models change with the seasons, and large-scale human experiments cannot be (ethically) conducted with the control as seen in physics. Because psychology experiments have substantial environmental variability, drawing robust conclusions in psychology is exceedingly difficult.
This fact should mean that very-high-certainty findings in psychology studies must be subjected to the highest scrutiny.
I think the implicit idea they wanted to sell was that a certain amount of positive thoughts creates a self-reinforcing psychological condition - making the grass greener in an endless cycle. You'd think that if this was true, religions, spiritual seekers and even random people would have discovered it independently many times over, and woven it into our collective and intuitive knowledge for thousands of years. But then again, the main figure behind this theory was a management consultant.
Even though it is after 2 days, I came back to this comment. I was reading Weapons of Math Destruction and it just dawned on me. Tbh when I first read your comment I didn't understand it but know I do. In the book the author talks about baseball stats while you gave an example about physics.
The more noise in the data, the less precisely you should claim to be able to measure it.
Social science data is notoriously noisy and erratic. Relationships are notoriously weak. If your numbers have a lot of precision, more often than not it's just meaningless noise not "real" precision in measurement.
Brown, N. J. L., Sokal, A. D., & Friedman, H. L. (2013). The complex dynamics of wishful thinking: The critical positivity ratio. American Psychologist, 68(9), 801–813. | https://arxiv.org/pdf/1307.7006.pdf
> “One could describe a team’s interactions as ‘sparky’ and confidently predict that their emotions would be subject to the same laws that govern the dielectric breakdown of air under the influence of an electric field. Alternatively, the interactions of a team of researchers whose journal articles are characterized by ‘smoke and mirrors’ could be modeled using the physics of airborne particulate combustion residues, combined in some way with classical optics"
To be honest I think this story is overplayed a bit. I've heard of that tipping point idea before, or the positive to negative ratio thing, but I think most in the field didn't take it more seriously than an observation that couples, groups, etc that function well are more positive and supportive.
It's good to call this stuff out but part of the reason no one else did is because outside of a certain subsubfield no one took it that seriously.
The whole point of science is to acquire knowledge that eventually gets communicated outside of the scientific body producing it. If the papers never leave the ivory tower, or never lead to other papers that do, it's just navel gazing.
When a scientific community has a chunk of their work that everyone in the tower knows is bullshit but never gets corrected, that metadata doesn't leave the ivory tower. What happens instead is that papers like this flutter out the window, get picked up by journalists who don't have access to tribal knowlege, and then get spread wide and far. The information appears legitimate because it did come from a trusted scientific body. It just had the necessary context that it's actually bullshit left off.
Every scientific community has an obligation to ensure that the papers they publish and advocate actually mean what they say they mean. If nothing else, because not doing that ultimately tarnishes the brand and undermines the trustworthiness of that community.
> Every scientific community has an obligation to ensure that the papers they publish and advocate actually mean what they say they mean. If nothing else, because not doing that ultimately tarnishes the brand and undermines the trustworthiness of that community.
Overplayed? It jeopardises the entire edifice of that subfield, and by extension psychology itself, if it is not vigorously disowned.
Sokal said it well (from the article):
> “For me, the real question is not about Fredrickson or Losada or Seligman,” Sokal says. “It’s about the whole community. Why is it that no one before Nick—and I mean Nick was a first semester part-time Master’s student, at, let’s be honest, a fairly obscure university in London who has no particular training in mathematics—why is it that no one realized this stuff was bullshit? Where were all the supposed experts?”
> “Is it really true that no one saw through this,” he asks, “in an article that was cited 350 times, in a field which touts itself as being so scientific?”
I think the answer to this is remarkably simple. You can align scientists and fields along a spectrum and group them into roughly two categories:
The scientists who make falsifiable theories and run reproducible experiments
-and-
everybody else.
On the left side of the spectrum is physicists working in long-understood areas, such as "predict the motion of elements of the solar system 100 years into the future using Newton's law and observed data", in the middle is "biologists explaining the behavior of C. elegans using a combination of molecular biology and machine learning", and on the far right side is "psychologists attempting to come up with universal rules about human behavior using studies with tiny n"
The folks on the far, far left have a fairly straightforward system that works pretty well. If you claim something like "FTL" or "new energy source", you describe your apparatus, publish your data, show people a demo in the lab, and if it's real, somebody else in the world will be motivated to replicate your experiment, or help show you why you're technically wrong. Almost nobody makes it far along this path by repeatedly publishing interesting, but provably wrong results.
In that middle zone, there's still an air of replicability, but important studies turn out to be really hard to replicate for dumb reasons, like "the temperature of the lab is off by 0.5C" or "you aren't using the exact same plasmid as mine, which I personally isolated when in <nobel prize winners lab> during my postdoc". It's actually quite easy to publish fairly wrong science, for quite some time, without anybody really noticing. For example, hundreds and hundreds of papers turn out to have copy-pasted gels, where the authors just made up results and published them and their conclusions. I guess nobody bothered to replicate that paper!
On the right side is psychologists. I don't even think this field, with the except of a few very limited areas, has the statistical power to make the strong conclusions authors come to, in 99.9% of papers. It's better to think of this area as people just sort of writing what they want to believe, and placing a pseudoscientific veneer on top. This would fall into the "not even wrong" category- it doesn't even really qualify to be evaluated as science. Or if it is, it falls into the category of "speculative complex systems theory".
You may well be right, but this is too casual an approach toward what purports to be science in a nominally good journal. Modern science is amazingly well insulated from “in-house” criticism after the review process. As a result crap can flourish. We need more data/analyst wasps.
The founders of that field got tens of millions of dollars for continuing that bunk research. You don't think that funding could have been used better?
Anyone have suggestions of the best way to call BS on clinical studies and their marketing? A publicly traded company's products are all based on BS science. I've pointed it out to them but they are dead silent. Just one example is them comparing (in marketing) the active group of one study to the control group of another study performed in different countries years apart with different inclusions criteria massively different n and insanely different dropout rates. It just blows my mind this stuff happens.
Complain to the agencies that are/should regulate them. FTC coverse false/misleading advertisements generally, but maybe just FDA for mediciney things?
Hi - I am the subject of this article. I don't know how to prove that other than tweeting a link to this post, which I will do right now (renewiltord already posted my Twitter account here https://news.ycombinator.com/item?id=30247543).
Thanks for the kind words to those who posted them. For everyone else, feel free to AMA.
I see this as an argument for adding more STEM to every degree plan. There's unavoidable crossover, and people need intuition to properly question unfounded claims.
Be careful what you wish for. Increasing the competence of fraudsters will simply make fraud harder to detect.
Had the original theory simply included a "measured" coefficient, the debunking would have been much weaker and never made it into a gotcha story for bystanders to enjoy. This was sloppy from the outset.
What we need isn't stronger technical skills, but strong self-correction mechanisms and incentive structures.
(Ironically, the management consultant who orchestrated this whole ploy is in the field of group psychology - so perhaps it was performance art all along)
True, but there is an upper bound to the amount of exposure one can receive before they are simply a STEM major, rather than another major with more STEM exposure.
But you shouldn’t be able to get a degree in a soft “science” without demonstrating that you can apply disciplined thinking by passing some proper STEM courses.
I think if you add more STEM people will use mathematical models incorrectly even more because they'll feel more confident about understanding the tools they don't.
> Additionally, Fredrickson references a textbook called Differential Equations, Dynamical Systems, and an Introduction to Chaos, a book Sokal says “is aimed at upper-level undergraduates in mathematics, physics and related disciplines.”
> “If she had studied 1/10th of what she said she studied,” says Sokal, “she would have seen right through Losada.”
The author making the bad claims seems to have a pretty decent math education. Or she's lying about what she studied...
Popperism is not, and has never been, the definition of science. Science is straightforwardly defined: it is a pursuit of research whose primary mechanism for gathering knowledge about the world is the scientific method. Difficulties in methodology aside, social sciences primarily use the scientific method, and thus they are science.
The whole "Cited 350 times" made me laugh. I used to think this type of number sounded impressive. Recently I just started to do a Masters and was told "1 citation for every hundred words", so dozens of papers need reading and it take for *ever* weeks to read just a small number. Then I recalled articles claiming most citations are never checked so, and you won't believe this, I skim read papers and if vaguely relevent inserted cursory comments from the paper and cited it. Did that go horribly wrong? Go on, guess. So 350? I so believe it.
Things like your master thesis or anything like that simply aren't included in the citation count at all, generally no one cares how much students or general public are citing your work so that practice has no influence on the metric being used.
As the article states, it was cited 350 times in academic journals, i.e. cited in 350 reviewed publications with novel scientific work written by her peers.
It was just a lazy metric to measure effort and engagement, and that relieved the lecturer of the need to read the paper himself to see if you'd absorbed it and analysed its relevance correctly. I can't say I'm unsympathetic - the guy had 50 students to mark. I'm guessing this is standard in academia but if they'd said "read and include relevant papers" I'd have done a good job on half a dozen and the lecturer probably wouldn't have been overburdened either.
> But in attempting to draw an equivalence between the physical flow of liquids and the emotional “flow” of human beings, Losada had simply lifted the numbers that Lorenz used in 1963 to explain his method in the abstract, numbers used merely for illustrative purposes. Losada’s results, along with the pretty butterfly graphs Brown had been shown in class, were essentially meaningless.
It was a bit crazy in the late 1990s. Chaos theory had been popularized, and the social sciences copied fashionable buzzwords and buzzconcepts from chaos theory, and used them totally out of context. Maybe it was crazy in the 1920s, too? Einstein's theory of relativity had become popular, and sciences that had nothing to do with gravitation, copied the "everything is relative" slogan.
I think they should have sent in their paper somewhere other than APA. Don't expect courtesy in consequential confrontations. I think similar about responsible disclosure sometimes. If a company played dirty in the past why bother with accomodating them.
I think they wanted to give the APA a chance to make the right call / improve, and it also seems they found the peer process valuable and they wanted to get that feedback.
Psychology in general is full of hucksterism. My sister got her bachelor's in Pyschology and she also found a whole ton of nonsense in it. She says that there's been a split in psychology and the real science was getting done on the "Biopsychology, Cognition, and Neuroscience (BCN)" side of the split while the psychology side is getting further and further away from real science and more into the realm of the made-up.
I did a psychology undergrad major in the late 00s. There was a shocking amount of bad science in the required courses. Lots of stuff that supported "common sense" but would be impossible to replicate, and the methodology didn't even align with the summary of why the study was important to learn about. Mountains of work built off imprecise self assessment tests. You can find a psych study to support just about any idea, and it seems like that's what many successful psychologists in academia have done.
Big part of why I decided not to pursue it as a career.
Seems he does a lot of this data policing stuff. Neat.
The funny thing is that a lot of people go around saying "yeah, engineers go around thinking they can just walk into field X and study it without context". Yup. They do, and they can.
>> The funny thing is that a lot of people go around saying "yeah, engineers go around thinking they can just walk into field X and study it without context"
I have a degree in Anthropology and transitioned into UX Research. 90%+ of what you do is observation in both fields. Most of the engineers and math people I've met are horrendously bad at observing behavior.
In some fields, its just a different set of skills engineers are not capable of applying very well.
>that a lot of people go around saying "yeah, engineers go around thinking they can just walk into field X and study it without context".
That's sorta the point of a degree in the sciences rather than going to a vocational school, right?
It's amusing that people find it arrogant. It is that they're showing a level of competence we're all claiming and maybe only pretending to have? Is it "swim in your own lane" union-style isolation? Is it sneering at nerds for not having the good sense to shut it? Is it inflated egos that still think the things they learned in undergrad were hard?
What I find sad about this is that the effort and skill required to debunk this nonsense is an order of magnitude higher than the effort and skill required to create it in the first place.
Aka, the "Gish Gallop" and other analogues, observations and quotes about this effect, made many times.
It seems weird this theory’s backbone was just some random person who contacted someone and said something like “hey I have math that says you’re right” and it just takes off from there. Until some other rando checks it out of curiosity…
"A Chilean business-consultant named Marcial Losada, 'who had begun to dabble in what had become his passion: mathematical modeling of group behavior,' sent her an out-of-the-blue email."
The article seems to ignore it, but Losada had a PhD in social and organizational psychology from the U of Michigan [1].
Nick Brown was a 50y/o CS/IT employee who wanted to cross-train to human resources and wanted "evidence-based stuff". He encountered lots of fantastic positive psychology claims that were being accepted wholesale as "quantitative". He traced the math behind a critical one back to a computational fluid dynamics paper, which the debunkees had used equations from with no context. (allegedly because of the use of "Flow" in positive psychology!)
He cold emailed professors asking for feedback, wrote a 3000 word critique, got help from a mathematician and psychologist in smoothing the writeup, and ultimately got it published in a top journal and the offending results recognized as flawed. The general crusade continues along with co-crusaders Sokal and Friedman.
The result in question was the legendary 3-1 positive to negative tipping point between flourishing individuals and languishing ones.
> It seemed a case of numbers fudging. In a valid fluid dynamics problem, the numbers plugged into the equation must correlate to the properties of the fluid being studied. But in attempting to draw an equivalence between the physical flow of liquids and the emotional “flow” of human beings, Losada had simply lifted the numbers that Lorenz used in 1963 to explain his method in the abstract, numbers used merely for illustrative purposes. Losada’s results, along with the pretty butterfly graphs Brown had been shown in class, were essentially meaningless.
Some make millions for Authors who take those papers and cite them in popular psych books. The actual authors of the papers are rarely, if ever, making millions.
I don't know whether it made millions, but Fredrickson did write a pop-psych book based on the critical positivity ratio: Positivity: Top-Notch Research Reveals the 3-to-1 Ratio That Will Change Your Life.
The 9 million dollars in grants that the author received probably were also completely wasted, since someone that would randomly apply CFD equations to people is unlikely to have published worthwhile research.
Remains to be tallied how many of her research assistants on those projects went on to have careers in the field... Careers which are now more or less tainted by the association.
Even harder to tally: How many of those deserve to be more tainted because they saw it for what it was but didn't speak up. (Yeah, I know, hard to do for a lowly assistant.)
Later in 2013, the critical positivity ratio aroused the skepticism of Nick Brown, a graduate student in applied positive psychology, who felt that the paper's mathematical claims underlying the critical positivity ratio were fundamentally flawed.[6] Brown collaborated with physicist Alan Sokal and psychologist Harris Friedman on a re-analysis of the paper's data. They argued that Fredrickson and Losada's paper contained "numerous fundamental conceptual and mathematical errors", as did Losada's earlier work on positive psychology, which completely invalidated their claims.[7] In their response, Hämäläinen and colleagues argued that there were no fundamental errors in the mathematics itself, but the problems relate to the interpretation and justification of the use of the model.[8] Losada declined to respond to the criticism, indicating that he was too busy running his consulting business.[6] Fredrickson wrote a response in which she conceded that the mathematical aspects of the critical positivity ratio were "questionable" and that she had "neither the expertise nor the insight" to defend them, but she maintained that the empirical evidence was solid.[9] Brown and colleagues, whose response was published the next year, maintain that there is no evidence for the critical positivity ratio.[10]
In response, American Psychologist formally retracted the mathematical modeling elements of Fredrickson & Losada's paper, including the specific critical positivity ratio of 2.9013, as invalid.
Since TFA is quite long and has quite an annoying layout, I tried Reader View in Firefox, and it gave me a completely different article. Weird. Thanks for posting the tl;dr.
Yup, same here. I think it was because it's one of those "infinite scrolling" pages, where when you've scrolled down to the end of the article another begins (and the URL in the address bar has also magically changed).
This is a fascinating story of how some questionable research slipped through peer review safeguards of academic publishing and flourished. More importantly, it shows how the scientific community eventually caught it and debunked it. The original results were recalled. Sadly, one of the main perpetrators slipped into murky waters of "business consulting" and continue to sell this snake oil:
" Dr. Marcial Losada, has a Ph.D. in Social and Organizational Psychology from the University of Michigan. Dr. Losada was invited to present his work at Cambridge University in England, Harvard Business School, Graduate Business School at the University of Michigan, Sloan School of Management at MIT, Kellogg School of Management at Northwestern University, Stanford Research Institute, and Institute for the Future in Menlo Park, California. He presented his findings on applications of nonlinear dynamics to team interaction and productivity at the prestigious Director’s Colloquium in Los Alamos National Laboratory. He has briefed Vice-President Al Gore and the president of MIT, Dr. Charles Vest, on the interaction dynamics of high performance teams."
I feel like the field and the journals in psychology have a history of being and currently are quite fragile. I have taken a couple of examples and done some simulations and come to the conclusion that the kinetic fragility index of the field is about the same as amorphous silicon dioxide most the time.
And as anyone will tell you if your building a house of that material. It is vital that you ensure no one around is looking critically at it. So you should paint it with a brick pattern, and ensure nothing heavier than a feather is ever thrown towards it.
In that regards it seems the journal in this case agrees, as they seriously resisted a scientific rebuttal of a piece until they realized they could not prevent it. Had they had integrity (which I believe we can model analogue to a high Young’s modulus, I haven’t yet finished simulations to give a precise number of how high it should be for the journal) then they would have been thrilled at the chance to adjust their building to not include the original material.
Interesting read! The most disturbing part is not fraud scientists, I think there will always be a certain percentage of fraudsters in any field. The most disturbing part is the 300+ citations, scientists who based their work off of an obviously garbage paper.
It seems inescapable that the reason why must have been complete mathematical incompetency across the entire field. Say a psychologist partners with perhaps a chemist and uses some organic chemistry to 'prove' some critical point in a paper. I could imagine brushing over the details and citing the paper when any chemist would see the fraud as 'obvious'. Such is the complexity of cross discipline science I guess, and the reality that the sum total of human knowledge is beyond any individual's ability to grasp.
That being said, knowing that some papers are utter garbage I should do my due diligence by taking a humble pill and reaching out to my colleague in the chem department across the street to tell me what this critical chemistry is about before I include it in my work.
> The most disturbing part is the 300+ citations, scientists who based their work off of an obviously garbage paper.
> It seems inescapable that the reason why must have been complete mathematical incompetency across the entire field.
I think you won't have to try very hard to find analogous failures to properly test the truth or falsehood of the literature's conventional wisdom in more "mathematically competent" fields.
Toxic positivity and toxic negativity are grating to some, myself included.
The issue is that soft, squishy not-STEM "sciences" aren't all that evidence-based or experimentally-verifiable. That doesn't make them completely useless, it makes them prone to fungible, inconsistent guidelines rather than hypotheses, theories, or laws.
> Losada had simply lifted the numbers that Lorenz used in 1963 to explain his method in the abstract, numbers used merely for illustrative purposes. Losada’s results, along with the pretty butterfly graphs Brown had been shown in class, were essentially meaningless.
This misuse and abuse of methods will continue as long as you can make a big splash in the field.
The technical correction of the misuse afterwards can then be tacked on to the end of your article later, with absolutely no recourse or penalty to your career.
If there was a collaborative project to review dubious published research papers, which of you would be interested in contributing? I think there are four categories of papers to focus on:
1. Papers with obviously false claims.
2. Papers that omit information that is crucial for verifying their claims.
3. Dubious papers that introduced ideas that attracted a lot of attention and subsequent sketchy papers.
4. Dubious papers published by people who just started in PhD programs.
Post-publish review is something that should probably be improved. It can't exactly be open to the public to contribute but equally the public should be able to see if scientists have (anonymous) serious doubts about a paper.
Unfortunately a lot of academic papers throw hopelessly tortured sentences, esoteric words, and complex formulas at you in the hope that no one notices that the whole paper is built on a house of cards: https://www.nature.com/articles/d41586-021-02134-0
You have to wonder how many people were harmed by this fraud.
And that's exactly what this is. The original paper was a fraud. No honest mistake can explain it. The reviewers committed fraud by not seeking competent input when they lacked the competence to do the math.
How many people under psychological care were given treatments and were harmed?
How many companies wasted on money on this and should be able to seek compensation?
And through it all, the three met only once in person, when Friedman was in London to give an academic address. It was a day spent scribbling equations in chalk on blackboards and talking strategy – a debunkers’ bonhomie.
I truly miss this about academics. Where you get a chance to sit down and think freely and pick someone else's brain in the pursuit of some hypothesis you're chasing down.
It doesnt seem possible to have a fixed positivity ratio when considering research on extinction.
The gist of extinction is that when training using positive feedback, the desired response will cease when the positive feedback does (extinction). In order to maintain the behavior the feedback has to essentially be randomized with a decreasing prevalence (positivity ratio).
I can guarantee you there are a lot of people really upset with him.
A lot of those kinds of studies are "this makes me feel good and this is the world I want to see" types. No consideration of the negative outcomes of bad info, bad practices, or even externalizes.
"The married father of two had graduated from Cambridge University in 1981 with a degree in computer science, and spent most of his career as an IT networks operator at an international organization in Strasbourg, France."
Yes, but the context was also with that person as a current student, which was not in CS, and his CS background didn't have much bearing on what he discovered. It would have been just as relevant to say "Father of two debunked..."
A better title would have been "New Student in APS debunks core finding in the field."
HN readers are predominately CS/IT folks, and would probably relate better to that aspect of his person than others. That's certainly why I read the article.
Should I emphasize that he lives in the UK?
"UK citizen debunks positive psychology"
Or should I emphasize the statehood of the person debunked?
"UK citizen debunks US professor with one neat trick"
Just trying to summarize the relevant parts of the story without clickbait -- which the article did pretty well I thought.
The relevant part was his status as a APP student, so that's what I would go with for the headline.
Using an irrelevant bit because it will get more clicks seems to fall under the "click bait" domain, especially when he was a grad from CS, not a current major.
Personally I would be more interested to know that it was a newbie student to an academic field debunked a core tenet. "New APP Students Debunks Core APP Finding" is much more interesting to me than a coincidence of the former academics pursuits with mine. It implies the question, "If a brand new student to the field saw this error so quickly, why didn't anyone else?"
Am I the only one who hates the phrase, " __ major"? One can "major" in a subject for all of three weeks. It's not some indicator of any sort of knowledge. At least the phrase, "___ graduate" implies that they at least learned enough to be issued a degree.
I got bored. This is an article style I really don’t like, the interesting information is buried towards the end after thousands of mostly irrelevant bits of narrative which aren’t worth remembering.
I can guess the ending though, technical guy sees psychologists appropriating a well defined mathematical structure, doing some hand waving, and claiming that it applies to your everyday life in a shallow way. Guy is hero.
Yes, psychology is absolutely full of bad science and bad scientists. Lots of fields are, but particularly psychology.
> Yes, psychology is absolutely full of bad science and bad scientists. Lots of fields are, but particularly psychology.
Social sciences in general are probably harder than other fields for the simple reason that it requires other people's time and effort.
A psychologist friend of mine once dropped the line "we know a lot about how university students view the world" when discussing the replication crisis. I think that while simplifying things quite a bit, it is very apt.
If the latter were true, I'd be surprised. It's obviously a quip about using students as subjects for credits, but the average psychology study reveals very little about students. After all, it's how they got convinced psi exists (Bem, 2011).
Economics is where you really glean insights into university students. For example, they enjoy free food exponentially more than they like very cheap food. Also, they great at playing games for money.
Western, Educated, Industrialized, Rich, and Democratic dominate the participants in psychology studies, even when they're not just psychology undergrads encouraged to participate in research.
It's more than that, people who aren't really interested in science or the math required to do good science are attracted to the "social sciences" in a much higher degree. People who are unqualified to do analyses are publishing papers and those papers are getting published with the approval of their peers who are similarly likely to be unqualified.
I'm glad I'm not the only one who felt this article was about twice as long as it needed to be.
I will throw out there that, despite all of the flaws and issues with Psychology as a field of science, it does seem like everyone involved, mostly, did the right thing here. I did skim it, but I didn't really see anyone do anything other than have sour grapes over how it went down, so I feel like this article and situation generally is a good example of a flawed system working out; deferring to truth over ego and orthodoxy.
Some groups would have circled the wagons to protect against this outsider, but that's not what ultimately happened here. Good!
>This is an article style I really don’t like, the interesting information is buried towards the end after thousands of mostly irrelevant bits of narrative which aren’t worth remembering.
This sort of journalism is something I find extremely frustrating. Honestly, I don't care about pretty much anything in the entire section about Brown meeting Wiseman. That whole thing could've been a single paragraph with no valuable information lost. Maybe this is just because we're in the ADHD age, but I couldn't read it because of all of this extraneous information.
This isn't an instant gratification issue, the details which pad this article just aren't very relevant to the point or the title. The life story and mundane details of this man's life don't, or barely, support the main point of the article which is how he debunked and got retracted a well-known paper in a field which wasn't originally his. I might want to read a biography of Albert Einstein, but the biography of this guy just isn't remarkable.
It's the same issue with SEO recipe articles where before the actual recipe there's an inane story about going to the beach with the family dog and finding a starfish before finally getting to the point about fish tacos, or whatever. If the article had been fine details about a few of the ingredients or sourcing them or preparation techniques down to specifics, sure, that would make sense; but if I ask you what time it is and you start telling me about how your grandfather served in WWII I'll probably be a little annoyed.
He traced the math behind it back to a computational fluid dynamics paper, which the debunkees had used equations from with no context. He cold emailed professors asking for feedback of his organized findings, got back a 3000 word article based on it, and with help from a mathematician and psychologist in smoothing the writeup, and ultimately got it published in a top journal and the offending results recognized as flawed. The general crusade continues.
The result in question was the legendary 3-1 positive to negative tipping point between flourishing individuals and languishing ones.
“In the last year, problems in reproducing academic research have drawn a lot of public attention, particularly in the context of translating research into medical advances. Recent studies indicate that up to 70% of research from academic labs cannot be reproduced, representing an enormous waste of money and effort,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “In my experience as a researcher, I found that the problem lay primarily in the lack of incentives and opportunities for validation—the Reproducibility Initiative directly tackles these missing pieces.”
https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
"Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias."
but I didn't really need to add that to make my comment any better.
I think it's pretty shallow still, even with that, because it's just a generic dismissal. Most readers are already familiar enough with this phenomenon that your comment amounts to a knee-jerk reaction—literally the most obvious thing one might say in the closest generic category. That's a marker of a bad HN comment, because it points to shallower, more generic discussion.
I mean, I could have made the point that this is basically just the base rate fallacy: because 90% of papers are wrong, finding wrong papers is easy. You can just say all papers are wrong, with 90% accuracy.
Or, I could have made the point that social sciences as a whole are woefully non-quantitative and the real lesson is: ignore just about anything you hear about psychology studies in the media, because most of them come from innumeracy and invalid experimental techniques.
Another great point: when somebody, say an "average Nick", finds that a paper does contain an invalidating flaw, it can be challenging to get the attention of the establishment. Sometimes that's because you're an average Nick with no cred in the field, sometimes it's that the journal doesn't want to admit that one of their board has been publishing total crap for the last 20 years in said journal.
Instead, I summed up all those points in a single comment which was in no way shallow, but rather, from decades of experience with our broken scientific system.
If you have decades of experience with the scientific system that's wonderful—but for a HN good comment you need to show that, not tell it—and you certainly need to make comments that are distinguishable from obvious internet dismissals. If a comment pattern-matches to that (as yours did), people are certain to interpret them that way, and not just moderators.
People with significant knowledge in their heads often assume that when they communicate a short message M, the message is substantive even if it doesn't include much information. In one's head, this is precisely true: M automatically connects with the things you already know and thus has a lot of depth to it. But all that is lost when sending M to other people—especially in stateless internet comments.
The rest of us don't have your experience and knowledge, and don't have access to what's in your head. All we see is the information you explicitly put into M. In this way, the same M can be both substantive (to the sender) and unsubstantive (to the receiver), because the sender has all the implicit information and the receiver only has the explicit. The 'received M' matters much more than the 'sent M', though, first because there are millions of receivers, and second because it's the received M that determines how people react.
I was all for writing long comments sharing my experience but the prevailing attitude on HN is "experts don't matter" and "covid vaccines don't work". No amount of additional contextual knowledge or explication will help those users (and I am exceptionally concerned by the latter).
I often extrapolate but sometimes, all that needs to be said is a single sentence, because it achieves my goal.
Ok, but for good posts, your goal needs to also take readers into account.
I'd be careful about any generalizations concerning the "prevailing attitude on HN". People routinely (I would almost say invariably) fall prey to false feelings of generality about this, because the annoying responses one encounters inflict a much greater impression on one's memory than anything else one encounters. I've written about this a lot, which may (or may not) be useful.
I think you must certainly have fallen prey to this bias, because "covid vaccines don't work" is extremely far from a prevailing attitude on HN. Such comments do appear, though, even though they're a tiny minority, and these no doubt make an outsize impression.
The movement he's part of is fundamentally disagreeable (in the 'big five' sense [0]), and some of its prominent characters, as you might expect, have reputations for being interpersonally difficult [1]. Collectively they've been labeled "data thugs" [2] and accused of "methodological terrorism" [3], which is why I think Nick's fundamental goodness is an especially valuable asset. We'd gain a ton of social value from certain reforms to the scientific research apparatus, and, IMO, being kind when delivering pointed critiques helps a difficult pill go down. That's different than pulling your punches; it just means staying focused on the thing you actually want to change.
[0] https://en.wikipedia.org/wiki/Big_Five_personality_traits
[1] https://medium.com/@OmnesRes/the-dumbest-fucking-thing-ive-e...
[2] https://www.science.org/content/article/meet-data-thugs-out-...
[3] https://statmodeling.stat.columbia.edu/2016/09/21/what-has-h...