Ego depletion is framed as an effect in a particular direction: applying self-control reduces self-control in the near future. So studies are called successful if they see that direction, otherwise unsuccessful.
But the other direction is a reasonable hypothesis too: applying self-control increases self-control because the brain regions are activated and ready, or something.
So you could frame the problem bidirectionally: applying self-control affects self-control ability according to some time-dependent formula with parameters that we'd like to estimate.
Framing the problem bidirectionally might reduce publication bias, since every estimate of the parameters would seem equally valuable.
The common mistake that many (but not necessarily you) make is to assume that conscious effort is proportional to energy use in the brain. The idea that willpower brain is some kind of store that can be depleted is not as parsimonious as other hypotheses.
Cognizing is a costly activity, whether daydreaming or thinking, a lot of stuff is going on. So why does thinking in particular feel difficult?
The least terrible analogy I can think of is to imagine a distribution network. Postal systems, railways, the internet; whether they are at 20% or 100% utilization, the marginal costs don't budge much. The other key concept is that the brain both learns as well as has to figure out how to do so efficiently. Reward schedules and surprises play prominently into this.
The hypothesis: some things feel difficult in a way not directly related to energy use because it's attention that is key and it is better thought of as a resource to be allocated and managed.
The questions are: why is it so hard to maintain focus on abstractions for extended periods of time? Someone is going to mention the ADHD hunter ancestor theory as selectors for extended focus, but even then, I will say that their attention still did not stay as fixed or devoid of obvious rewards as some abstract thinking requires. Is focus trainable? Why are some better at not responding to immediate rewards? Is this trainable?
I think conscious effort in itself is not difficult. What is difficult is to be lacking acceptable options or strategies to solve the problem at hand. That is causing anxiety and fatigue.
To put this in terms of machine learning, the brain is doing reinforcement learning maximizing a value function. But when the value function is very low and options (strategies) available are too limited, the system can't output a satisfactory action for the next step so it interprets this as suffering.
It's all about availability of choices, not the actual effort. What's depleting is not energy in the brain but the perception of how well we can handle the situation. The state of flow is also related to conscious effort, but in flow there is always a good outlook, there are many options available, many strategies, creative possibilities of combination, and consequently, it's a joyful activity.
That's a very interesting point. I had never thought that conscious effort would be proportional to energy use, as if some tiny gas tank in the brain would run low.
I can happily move from one high-focus thing (e.g., coding) to a different one (e..g, complex game) without problem. And as long as a problem keeps being interesting (that is, providing novelty or progress) I can focus for very long periods. That made me think that my experience of mental tiring was more a way to prevent lock-in and to keep me away from situations of diminishing returns. In our natural environment, I'd expect the cost/benefit curve for pure abstract focus to drop sharply, so it wouldn't surprise me for there to be a special mechanism to get us out of our heads and back into dealing with our environment and bodily needs.
I understand focus is related to dopamine, a neurochemical and it is know that nearly all neurochemicals can be depleted or their pathways become less effective because of various run-time compensations that occur.
Coffee effects me most for the first cup, most likely because I have stores of adrenaline and dopamine, after than first cup any cup after than decreases its effect on me. The first coffee probably is more noticable because of the difference between my no coffee state and my coffee state -- which is difference than a wearing off coffee state and a free cup coffee state.
It is reasonable to be able to measure these effects. Not saying this effect is true, but neurochemical store depletion as well as real-time and long-term compensation are real.
The effect might be modal, depending on other variables (expectation, sleep, diet etc). So it works one way on one person or at one time, and differently elsewhere. Our brain states are likely fantastically modal, since its an enormous state/storage unit.
This really strikes me more as a problem of incentives than "lol psych researches need to learn statistics".
Incentives in this situation are funding and publication. These both require positive results. This seems to be tied to increasing monetization of science driving it away from a principled search for the truth.
We also see this with patents. My girlfriend has her first lab job this summer, and they had a day long seminar with the university's patent foundation on methods to ensure that nobody stole their ideas. This leads to a lot of secrecy (for example, this particular patent office doesn't allow researchers to store anything in the cloud, which speaks to lack of technical understanding, but also to an intense paranoia). This also dramatically reduces the capacity for sharing basic lab know how - there's nothing remotely like StackOverflow for scientists.
This issue is a bit near to my heart, as the company I work for was originally founded to help researchers share data in a secure way. It turned out that there wasn't any demand for that, because, as our CEO put it, "Everybody thinks their going to win the damn Nobel Prize". So we pivoted and now sell mostly to financial and tech companies that are serious about protecting user data.
The main thing I got out of the article was that I should ignore results from the fields of psychology or sociology for the foreseeable future. The standard of proof seems very low.
I never fell for Tony Robbins-type pop psychology, but I loved books like "Influence: The Psychology of Persuasion" by Robert Cialdini and "The Tipping Point" by Malcolm Gladwell.
I thought I was getting useful, non-obvious, actionable information. I'm disappointed that the lessons from those books might be BS as well.
> I should ignore results from the fields of psychology or sociology for the foreseeable future
Yes, you should. The problem is that some of these bad results have become key political tenets. Try voicing skepticism about the reality of stereotype threat and see how long it is before HR finds some excuse to fire you.
If you haven't noticed, there are powerful social pressures in the Bay Area to go along with certain kinds of reality denial. The more reality suggests that dominant Bay Area social beliefs are false, the louder supporters reiterate these beliefs and the greater the ferocity with which they punish dissent.
I didn't know the term "stereotype threat" and had to google it. Here is one of the clearer explanations I found:
"Black and White college students a half-hour test using difficult items from the verbal Graduate Record Exam (GRE). In the stereotype-threat condition, they told students the test diagnosed intellectual ability, thus potentially eliciting the stereotype that Blacks are less intelligent than Whites. In the no-stereotype-threat condition, the researchers told students that the test was a problem-solving lab task that said nothing about ability, presumably rendering stereotypes irrelevant. In the stereotype threat condition, Blacks did less well than Whites. In the no stereotype-threat condition, Blacks' performance rose to match that of equally skilled Whites."[1]
"Merely telling women that a math test does not show gender differences improved their test performance. The researchers gave a math test to men and women after telling half the women that the test had shown gender differences, and telling the rest that it found none. When test administrators told women that that tests showed no gender differences, the women performed equal to men. Those who were told the test showed gender differences did significantly worse than men, just like women who were told nothing about the test."[1]
Which shouldn't be surprising, because unless carefully expressed, skepticism towards the stereotype threat effect can be interpreted as a justification for stereotyping, especially when the skepticism is expressed in the context of political tenets.
Stereotypes are accurate, though. People are good pattern-recognizers, and the characteristics that we use to form stereotype do convey useful information.
Pretending that stereotypes are factually bogus is wrong. It creates resentment. It's much better to accept that stereotypes exist for a reason, but emphasize that it's a grave injustice to let stereotypes hurt our evaluation of people who defy these stereotypes.
> skepticism towards the stereotype threat effect can be interpreted as a justification for stereotyping
Current HR policies go beyond requiring fair treatment of individuals as individuals. Every sane person supports doing that. HR demands that we silence truths about the natural world go unstated. HR demands that we make untrue statements about the world. That's anathema to anyone technically-minded.
I like to think of myself as technically minded. However, your stereotype of technically minded people seems to sugest that either I accept the "natural truth" of stereotype accuracy or forfeit my technically minded view of myself. I will do neither.
The claim that one can engage in stereotyping people while exibiting fair threatment of the same people as individuals is dubious to say the least given the definition of stereotyping.
Once engaged in stereotyping no amount of post factum fair threatment of the individual will make up for the unfairness.
Later edit:
People are imperfect pattern recognisers, in fact our pattern recognition generates a significant amount of false positives. From an evolutionary perspective this was acceptable at a time when tigers could be lurking in the bush. The penalty for running from an imaginary tiger in the bush is insignifiant compared to the penalty of not running from an undetected tiger.
As a layman when it comes to statistics and academia, this is the first I've heard of "publication bias". It seems like a pretty big problem at first glance, especially if it recurses and calls into question meta-analysis being done to investigate the presence of the bias itself. Sounds like a massive headache.
If this problem is as big as it sounds, what are the repercussions?
Think about it this way: Suppose my theory is that a fair coin is not actually fair but always lands on heads.
If I flip a coin one million times and only publish the ~500,000 results that show coin toss coming up as "heads," I could lead people to believe that a fair coin will always land on heads.
That's how misleading this bias can be.
But anyone can toss a fair coin, so I'd be found out pretty quick. But with more complicated and murky stuff like psychology this can be a huge problem, especially if people begin using this shaky work as foundation for things like: public policy, how to live their lives, how to do work, etc.
After all they can't just toss a coin to check my work on the more complicated stuff.
To take your fair coin example, publication bias is when the only studies that get published are the ones that show an unfairness in the coin – the rest of the studies that simply find 50% heads and 50% tails (aka a fair coin) don't get published because they're not interesting.
Fraud requires intent. Now, this extreme of bias would typically require said intent, sure. But let's say the experimenter has anterograde amnesia.
They get an idea, flip a coin - oh hey, it's heads! They publish the interesting result. They get an idea, flip a coin - oh, it was tails, nevermind that theory. They get an idea... each time, they don't remember that they've already attempted the theory. No intent, no fraud, just bias.
The good news is this is a pretty unlikely extreme in an individual. Anterograde amnesia is rare. The bad news is nobody can remember that someone else attempted a theory and came up with a negative result that they didn't bother to publish, and even individuals will have a hard time perfectly accounting for and preventing their own bias within a single study. The even worse news is that fraud exists as well.
This doesn't seem very feasible given the current level of privacy afforded to companies.
I don't think there's a way to distinguish between studies that have lost funding because there are genuinely less funds to go around, and foo experiment just happened to be one of the ones that didn't make the cut, and studies that have had the funding pulled for harmful reasons (Like negative results).
Mainly because it's devilishly easy to mask; you'd just reduce funding to research and redistribute it into something plausible like marketing. Of course this isn't feasible for lots of studies producing negative results.
Of course, one way to counter that is to insist that the data is published regardless, but I'm sure that the data could be hidden with clever use of NDAs and court arguments along the lines of "We need to prohibit the distribution of company assets".
> Of course, one way to counter that is to insist that the data is published regardless, but I'm sure that the data could be hidden with clever use of NDAs and court arguments along the lines of "We need to prohibit the distribution of company assets".
I find myself more and more treating complexity as a proxy for dishonesty. Companies that are trustworthy seem to have a relatively simple business model that they don't try to hide. The more convoluted it is, the more likely it is someone is scamming you. Wonder if that could be used as an effective metric - the more complex the reason someone weasels away from publishing the data, the more their "reputability" score goes down?
What about starting by implementing these measures in public universities? That seems very feasible, and could have a snowball effect on how studies are broadly analyzed and perceived.
The repercussions are pretty depressing for social sciences. Especially Psych, which relies on a lot of "30 undergraduates were recruited" to demonstrate effects. Particularly damaging is that this particular theory (thinking hard leads to drops in blood glucose levels) was widely promoted in the general public; I recall watching a TED talk about it, as well as a pop-sci book.
There was an econtalk episode late last year with Brian Nosek about the Reproducability Project (http://www.econtalk.org/archives/2015/11/brian_nosek_on.html). They went the further step of analyzing the wider field to see how bad the biases are. Previous attempts to call out replication (priming theory) did not go well, leading to a lot of hostility. Nosek's process is a bit more friendly, by inviting the original authors's critique early, before data collection. I figure it's a bit harder to blast the auditors for considering a replication study without appearing as though you doubt the effect is real yourself. They still got a bit of pushback, but nowhere near as acrimonious.
It sounds like some of these researchers need their own ego depleted, for the sake of science. Critical, well-intentioned analyses of a study by other groups should be a welcome thing, as it will lead to more robust results. It seems like the incentives have gotten way out of whack.
Bear in mind it's not just ego, but livelihood. People's grant lines and tenure cases rely in part on contributing to the development of the field, and having the basis of your research retracted usually undermines your tenure review and puts a hamper on continued grant funding. So it's not super surprising that people are defensive, but I'm always amazed at how publicly aggressive things can get.
Oh it's far worse than that. Iirc about a third of studies fail to replicate, and in psychology it might be higher. In fact we even have a control group for science. A fake field of research that spends it's time studying phenomena which can't possibly be real, the paranormal researchers. They produce tons of positive results anyway. See The Control Group is Out of Control: http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
If your view is a priori that psi effects are impossible all you will see in the studies that 'it must be invalid and flawed or faked'. Yes, this makes people doubt scientific practices in general. You will always feel you are winning that argument, no matter what criteria those scientists meet.
But psi researchers these days jump through far more hoops, run far more replications, spend far more time looking for weaknesses and alternative explanations, garner far more scrutiny and second guessing. To call that a 'fake field of research' when it is more rigorous is disingenuous.
Negative results do get published regularly. And more replications take place. It is quite hilarious really, because it makes people panic about what is wrong with the methods we apply across the board in science. Skeptics just raise the bar abritrarily and shift the goalposts as a means of sweeping things under the rug while calling it scientific rigour.
We have also now gotten to the point that the fact that we commonly have failed replications (across all social science), it means the whole premise of a line of research must be completely flawed. But we should expect failed replications - for one, it's impossible to repeat a set of circumstances exactly. Many replications aren't true replications.
> But psi researchers these days jump through far more hoops, run far more replications, spend far more time looking for weaknesses and alternative explanations, garner far more scrutiny and second guessing. To call that a 'fake field of research' when it is more rigorous is disingenuous.
But that's the point of Scott's article. We have more than enough reasons to believe the whole field is bullshit (did you see a working practical application of that research?), but because it does all the moves and rituals of Real Science, it forms a perfect baseline for how many positive results we can expect with pure cargo-cult science.
Explaining away thousands of studies that happen to study controversial effects with unknown mechanisms in one brush is not demonstrating intellectual rigour nor does it make you come across as upholding 'real science'.
The explanation that psi-researchers are just good at finding the appearance of signal from noise, small as it may be, is fine, but then personally, I'd like to see the same vigour for explaining what the signal actually is. And people that have a firm belief that any 'psi'-explanation can't possibly true (and this a pretty arbitrary notion at that, as psi would just follow conventional physics and adhere to normal biological mechanisms) just aren't going to accept any other explanation. So they look for issues with the statistics used.
The notition that the whole field is BS, as you put it, requires all the researchers in involved to be utterly mad, stupid or deceitful. That is a claim easily made by skeptics, a thing that they have no trouble agreeing to. That view is not born out of scientific honesty, it's just another form of cargo-cult thinking.
The point of the linked article is exactly that psi researchers go through far more rigorous methods than most researchers. The fact that they still come up with wrong results tells us just how far the goal posts need to be moved - but not just for psi, for all of psychology and other fields too.
> But psi researchers these days jump through far more hoops, run far more replications, spend far more time looking for weaknesses and alternative explanations, garner far more scrutiny and second guessing. To call that a 'fake field of research' when it is more rigorous is disingenuous.
So what's a good study you can point me at? The nice thing about psi testing is that you can get a result sample every few seconds. It should be easy to turn even a moderately small effect into a five sigma result that's immune to publication bias.
"During 2013 and 2014, a total of 1479 people from 77 countries contributed 2985 test sessions. Over the same period, 5738 sessions were run as controls by a computer programmed to simulate human participants. The results showed that with human observers the fringe visibility at the center of the interference pattern deviated from a null effect by 5.72 sigma (p = 1.05 × 10−8), with the direction of the deviation conforming to the observers' intentions. The same analysis applied to the control data resulted in an overall deviation of −0.17 sigma."
There are some presentations on Youtube that go into the details of the double-slit line of experiments.
In medicine all kinds of things were done to avoid publication bias (most notably trials registers). It still is a big problem, because the countermeasures aren't enforced. In most other areas people don't even try to counter it, which means it's probably much worse and we don't know how bad it is. It's a major reason why so much of science is most likely false.
I suggest you have a look at the work of Ben Goldacre, he's been a major voice in highlighting this issue. He's done a ted talk introducing the problem and later has written a book (Bad Pharma). He's only focussing on medical trials, but you always have to think: "This is happening in all areas of science".
It is considered a serious problem by many, being a key part of the whole replication crisis. The idea that studies need to be preregistered is slowly gaining ground I feel.
It seems to me, not serious enough for decisive action apparently. Progress is slow on that front. "Knowing" and "doing" are different. Most people who smoke or eat badly are aware that it's a (serious) problem, but it does not necessarily lead to action. So that people "are aware" of something doesn't tell the whole story.
While the picture seems bad for Ego-Depletion, I find the counter arguments put forward in the article not wholly satisfying, either.
So they tried to replicate it in a large effort and failed, but they didn't use the same test as in the original paper. How is that a replication study? I think to properly fail a replication study, you should use the same test as in the original study. In the replication the used many participants, but developed a cheaper test to administer (wholly computer based).
As it is, they have another indicator that Ego Depletion might not be real, but just as well there could be some other effect that explains why the replication failed.
Likewise for meta analysis: it looks good on paper, how could an average of n studies be wrong? Surely the mass of papers studied will cancel out all the flaws of the individual studies? But on the other hand it seems to me meta analysis would overestimate the impact of bad studies, as they are easier to make so there a probably more of them.
A meta study is certainly an interesting data point, but I feel it can not always be the end of the story. It can maybe point out the need for deeper questions or further research, but not be proof on it's own.
By the way, the marshmallow study has been redone and reinterpreted. It has been found that the children's behavior is heavily influenced by their life experience. When they live in an environment where they can't trust adults they eat the marshmallow sooner, quite rationally. So it isn't an inherent trait of the child but of its environment whether it waits for the promised future return or not.
I've seen an interview with a researcher who said something along the lines of "we take reliability into account" - but what he then said seems to show a very narrow interpretation. The experimenter must be well-known to the children and has played with them before the experiment, and that "the marshmallows are right there in front of them, so it's not a reliability issue". Personally, I don't think that is even remotely enough to account for the effect and that is a somewhat naive take on the issue, especially the second part that solely relies on reason. Which doesn't even work in adult scientists to counter social effects.
Run an n=1 study on yourself. Even if it's full of placebo effects and doesn't generalize to anyone else, as long as you figured out what makes you feel/do better, you win.
But the other direction is a reasonable hypothesis too: applying self-control increases self-control because the brain regions are activated and ready, or something.
So you could frame the problem bidirectionally: applying self-control affects self-control ability according to some time-dependent formula with parameters that we'd like to estimate.
Framing the problem bidirectionally might reduce publication bias, since every estimate of the parameters would seem equally valuable.