Rather than publish all of the dubious speculations and the analysis whose problems took time to emerge, simply publish the data. The advantage, in his words:
> In the example of the ESP study, if anything’s valuable it’s the data. Publishing the data would get the journal off the hook regarding fairness, open-mindedness, and not missing a scoop, while enabling others to move on reanalyses right away, and without saddling the journal with an embarrassing endorsement of a weak theory that, it turns out, was not really supported by data at all.
Incentives: Spend five years doing difficult / expensive / dangerous research, gathering data that demonstrates X. The data set is vast, and you can get ten papers out of it, one every six months with some awesome collaboration and possibly tenure.
Do you do that or
publish all your data in the first paper and watch a dozen grad students publish your next nine papers in three weeks ?
Incentives matter yes. Science and data should be free and transparent. But if we pay peanuts and expect the monkeys to appreciate the applause of publication, we need to change our incentive structure.
The reasons AI / ML researchers don't mind publishing the data early is because they already get 500k salaries and equity.
Then there is a huge problem with how we publish Science... Decades ago, sharing all these huge datasets would have been quite difficult if not uneconomical.
But nowadays there is no reason left for not publishing the data and hopefully get academic rewards for that...
I have seen huge amounts of data not being digitized and lost because of this ownership mentality...
It is not a 'mentality' issue. It's a systemic issue. Researchers need to eat too, and it's already difficult enough that few are willing to sacrifice much for the sake of morals.
publish all your data in the first paper and watch a dozen grad students publish your next nine papers in three weeks ?
The way I would expect this to work is, the grad students give you credit for gathering the data, which allows you to share in the prestige associated with all of their projects.
That's like someone releasing a CGI TV series called "Woody and Buzz in the toy store" with exact copies of the digital models. And not expecting Disney to say "we spent 1 BN promoting this brand, we expect a cut / control / story approval / your heart on a plate " [#]
[#] I believe that last one is in actual fact the Disney legal teams' opening gambit in all copyright negotiations. Then the gloves come off.
If graduate students can publish papers based on your data three weeks after they first see your data, then surely you are slacking off if you can only publish one paper every 6 months.
More plausibly, if your data is that apparent,
- you publish your first paper with your data, this takes 6 months from submission to publication
- you submit a paper every three weeks (also with your data) during that 6 months. that totals 8 papers before your article and data hit the press
- after your paper is published, it is 6 weeks before anybody fully gets the data
- during that time you submit your other two papers
- the grad students you mentioned now submit 8 papers, but are forced to cite your work
You now have 10 papers submitted before the competition and have 10-20 papers being submitted that cites your data and work. Total elapsed time = 14 months.
sorry, citing by other people do not have value equal to publish article on first author... and just do experiment should spend very very time and money, e.g., "if I add this compoment to cultured cells" just spend one day and then spend half of year to add component to cell each day and collect results...
You seem to be forgetting that many western groups are very small: 1 senior (often not even tenured) and 1-2 grad students, with not even a lab manager in sight.
In those conditions, going slower than a 50-grad-student chinese lab is not slacking off.
There is an incentive against doing it. No researcher wants the embarrassment of having an error that they didn't know about discovered. Journals don't fancy the embarrassment or hassle either. It would serve broader scientific progress, but that's not who is deciding to hide the data.
It follows from your comment that we should have anonymous authors in addition to anonymous reviewers. Can a researcher establish a provable track record of doing quality work while remaining anonymous?
Presently, for sure. I was wondering in the hypothetical, too, though. Authors really just need to provably show that they have a strong influence/presence on the publication graph or something. Although, reducing a researcher's entire value to a single scalar index is probably fraught. Also, egos would never go for completely anonymous authorship system, I think.
There's likely multiple forces, but at least one must be money. Someone's gotta host the data, right? That falls on either the universities or the publishers.
The approach fails miserably when taken in aggregate. I would read five to ten papers from my own field a week while in grad school. If I had to do the analysis for myself as well there wouldn't be any time to do my own research.
Add on twenty more papers a week from adjacent fields where I don't even know all the techniques and the magnitude of the problem hopefully becomes clearer.
Gelman's suggestion is how to handle articles with unlikely conclusions: don't publish the submitted article, though that presumably would be circulated as a preprint, but just the data. Then the journal is bringing attention to interesting data without endorsing the dubious analysis or conclusions.
> Pre-registration only assures that your study will not get any worse than it was the first time you thought of it, which may be very bad indeed.
>
> Searching for publication bias only means you will get all of the confounded studies, instead of just some of them.
No, pre-registration assures that when you do a study and get a negative result, you don't simply forget to publish it - helping to reduce publication bias.
But also, it prevents you from changing what you were looking for (because that's part of the pre-registration data) and "data dredging" for something that looks like a positive result, helping to reduce p-hacking - specifically of the "Texas sharpshooter fallacy" variety.
What you wrote and what the author wrote do not conflict. Indeed, the author already noted the goal of preregistration earlier in the essay. He is explaining here why it doesn't save you from different issues.
How does pre-reg handle cases where unexpected findings are made along the way? Are those data to be scrapped so everyone has to start their pre-registered "real" study from scratch?
The "fits of nervous laughter" paper by Wiseman & Schlitz (1998) was replicated by Wiseman & Schlitz (1999) [1], with a similar yet different result: RW (again) found no effect of staring, MS found a significant (p=0.05) effect of staring, but with opposite direction than the 1998 paper.
> That is, in let’s say a drug testing experiment, you give some people the drug and they recover. That doesn’t tell you much until you give some other people a placebo drug you know doesn’t work – but which they themselves believe in – and see how many of them recover. That number tells you how many people will recover whether the drug works or not. Unless people on your real drug do significantly better than people on the placebo drug, you haven’t found anything.
I had a drug program in which we blew away the standard placebo rate … unfortunately our placebo arm did too, though not by as much. But it was just enough that we weren’t enough better than our placebo arm.
We changed modality from the current standard (we were an injection rather than a tablet and all the trial participants were super excited.
This is something we know to be true already. The more invasive, the more powerful the placebo effect.
>Weirdly enough, surgery’s invasiveness may explain some of its potency. Studies have shown that invasive procedures produce a stronger placebo effect than non-invasive ones, said researcher Jonas Bloch Thorlund of the University of Southern Denmark. A pill can provoke a placebo effect, but an injection produces an even stronger one. Cutting into someone appears to be more powerful still.
Hmm. What happens when patients are given general anesthesia and just told they've had invasive surgery?
What if you supplement with a superficial incision and stitches, or a subdermal injection that causes some localized edema (and a dressing that obscures the fact there was no incision)?
For that, they would have needed a control group that took placebo pills. They can estimate the likely behavior of such a group from other trials, but it's not a rigorous experiment unless they do it themselves, carefully controlling all the other possible variables.
You can't do that, I'm afraid. Too many uncontrolled variables still. What if placebo rates change seasonally? Also, you cannot properly double blind the test - the experimenters all know about the prior study, and know which cohort they are treating.
To be fair, your attitude is quite common in cash-strapped university departments - but it leads to bad science.
For the particular indication we were going after there had been a number of topical trials and academic studies that showed that the placebo response was less than 1%, essentially no different from spontaneous (i.e. no treatment). Same was true of oral. This is pretty common in fungal infections unfortunately.
The FDA wanted us to compare against the topical treatment placebo rate for various technical reasons.
Perhaps you thought that “standard” was some sort of standard for any placebo for any indication and any mode of administration? Indeed that would make no sense.
Even specifying a placebo itself is non obvious. Of course your procedure and the object itself (tablet, whatever) is ideally indistinguishable by the study participant and, ideally the clinician (hard to do, say, when it’s a surgical procedure; you can’t hide that from the clinician). But even with a pill, you can’t always just supply, say, a slug of mannitol — that might actually affect what you’re studying.
Thanks, this is helpful additional info. Sorry, I didn't mean to contradict you, I was actually trying to reinforce the principle behind your experience. There is no true "standard" for what sort of placebo should be used in a trial, even if something has become "standard" in the sense of commonly used. It is not something that can be specified in the abstract, but is a matter of expert judgment with many variables.
Your trial gave an appropriate placebo, regardless of the "standard" in the field, and thus you were shielded from overstating the evidence of drug effectiveness.
So maybe that isn't what the GP meant? You're telling the person who did a study that they misunderstand the utter basics of studies, based on a 3-sentence comment they wrote.
Sometimes people misunderstand the basics of their full time work yes
We've all seen that person, and almost all of us have at times _been_ that person
And unfortunately, given the replication crisis, there is no reason to assume that a published scientist is not making a fundamental error that undermines the validity of their work. It apparently happens quite frequently, and we should be quite grateful for those that question and probe at assumptions and potentially faulty analysis techniques
Arthur Conan Doyle, through Sherlock Holmes, said, once you've eliminated the impossible, what remains, no matter how improbable, must be the truth.
This tells me that Schlitz actually has psychic powers and Wiseman does not. It actually makes perfect sense. People who are psychic wouldn't be skeptics, and people who are not psychic certainly would be skeptics.
I suppose that, had Conan Doyle been aware of quantum mechanics, Sherlock Holmes would have spent a portion of every case entertaining the notion that quantum phenomena caused the universe to spontaneously arrange the crime scene.
Holmes was famously ignorant of any knowledge that didn’t help him solve crimes. In his own words:
“You see,” he explained, “I consider that a man’s brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things so that he has a difficulty in laying his hands upon it. Now the skillful workman is very careful indeed as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. It is a mistake to think that that little room has elastic walls and can distend to any extent. Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.”
“But the Solar System!” I protested.
“What the deuce is it to me?” he interrupted impatiently; “you say that we go round the sun. If we went round the moon it would not make a pennyworth of difference to me or to my work.
Given this, I’m not sure Holmes would bother with a quantum viewpoint of the world, since the theory wouldn’t help him solve crimes.
My comment was a bit tongue-in-cheek, but if we're going to push our glasses up and engage in some good old well-actuallying, I'll note that Conan Doyle appears to have retconned Holmes' self-professed ignorance later on, as he calls on knowledge of astronomy to solve a case at least once: https://adsabs.harvard.edu/full/1993JBAA..103...30S
Perhaps the leading sentence by parent was not a true speculation, just an elaborate setup to the slightly altered quote as the punchline. In an alternate universe, if Conan Doyle took more interest in the digestive system of the human body, Sherlock Holmes would've explained everything with gut feelings instead of making extended retreats to his Mind Palace.
If you don't know it already, you might be interested in 'Dirk Gently's Holistic Detective Agency' by Douglas Adams and/or the funnier second part 'The Long Dark Tea-Time of the Soul'.
> This tells me that Schlitz actually has psychic powers and Wiseman does not.
This sounds like fertile ground for further research. What are the boundaries of these powers? In what circumstances do they work, and when do they not? What if the trial participants had all been greeted by a 3rd person, not Schiltz himself? What if he were wearing gloves, or a hat, during the trial? What if the participants were?
I'm not making fun- wouldn't such experiments go towards discovering how such observed powers work?
> This tells me that Schlitz actually has psychic powers and Wiseman does not. It actually makes perfect sense.
Or that exposure to Schlitz's and Wiseman's pheromones activate/deactivate subjects' psychic powers respectively.
> People who are psychic wouldn't be skeptics, and people who are not psychic certainly would be skeptics.
Skeptics and psychics spontaneously form self reinforcing social groups! Blah blah gender of researchers affecting mouse studies blah blah women's menstrual cycles sync up in small communities blah blah metronomes blah blah spooky action at a distance blah blah many worlds blah.
Honestly this XKCD arguing that psychic phenomena don't exist because they're not exploited for profit is a comforting difficult to refute argument for me: https://xkcd.com/808/
The author of this blog, Scott Alexander, has an online novel called Unsong [1] based on this idea. In the book's universe, Kaballah is real and (among many other wild things) corporations automatically generate long lists of Hebrew characters to find names of God that give various powers.
> In the book's universe, Kaballah is real and (among many other wild things) corporations automatically generate long lists of Hebrew characters to find names of God that give various powers.
In this universe, we do the same with numbers generated by ASICS or GPU: it mostly gives the power of money.
It may be that psychic powers exist but that they are not powerful, general, or controllable enough to be reliably profitable.
Also, if every profitable phenomenon in the world is already developed, then no business should ever invest in fundamental research - i.e., the pharmaceutical companies are wasting their research dollars.
Why is it comforting to you to believe psychic powers don't exist?
Yeah, e.g. someone who can, for example, consistently predict single bits of output from true random number generators 51% of the time would be evidence of something very strange going on, but not necessarily usable for anything practical. (Maybe if you could gather enough of these people to overcome the "weak" ability?)
Fair point, maybe it applies more to categories like dowsing, where doing marginally better than chance (or general unaided human ability to detect whatever) would have huge implications, but might not be profitable to employ when compared to modern technological methods, and definitely doesn't allow for easy reinvestment. Who wants to rely on a dowser who is wrong often and can't explain why when you could have a geological survey done that's more reliable and explainable?
what's the point in a dowser that's wrong often? isn't that like having a "slightly better than chance" super power? ie, can't fly but can jump higher than average (some times)
It depends on what you mean by 51% of the time. If you’re just making a directional prediction, “will the next market day be up or down?”, that’s nowhere near enough.
A little over 53% of days are up (green) days and yet you can’t use that 3x edge over a 51% predictor to make all the money.
I don’t have access to tick-by-tick feeds any more to test this, but I’d predict that “X will go up or down next second, [ignoring UNCHs]?” is predicted more than 51% of the time by predicting the same trend as the current/previous second and that you’re going to win on percentage of guesses and lose on money (because the changes in direction seconds [where your guess is wrong] are higher vol than the trend continuation seconds).
Again the difference is that everybody (who cares enough) know about the trends, but only you know about the 51% predictions.
Assuming these information sources are independent - even if the trends predicted the market in 99% and your prediction was only 51% - you can still combine them to get a better estimate than the generally known one.
Wait, what? I don’t have a specific cite, but I thought that predicting a random number generator that well is good enough to break common crypto algorithms?
IIRC, doing better than chance by 2^-32 is enough for the randomization system to be thrown out, and marching the output 51% is way better than that.
But yes, I’d agree with a steelmanned version of your point with a much smaller improvement over chance. But then, you would have to do a realllllly long test and it wouldn’t actually look that impressive.
Thanks, replace 51% with (50 + small non zero)%, anything more than chance, assuming the RNG has no bias to exploit, is "doing the impossible" but not necessarily feasible to profit off of.
Also, certain supernatural abilities would have been so useful in stone age societies that they would have given an evolutionary advantage to the individuals possessing them, and hence we all should have those abilities by now, as evolution should have made those adaptations more common.
In Larry Niven's "Known Space" books [0], there was a planet where telepathy was common. Natural selection kept it in check so a hunter would only be able to nudge a prey into walking into an ambush. Then an alien exploratory vessel lands on the planet, aliens that have no resistance to telepathy are instantly controlled and you get an intergalactic slaver empire from a neolithic culture in one generation.
So red queen effect [1]. Maybe telepathy evolves in tandem with resistance to telepathy.
The phenomena discussed in that comic were chosen for inclusion specifically because they actually would lend themselves quite easily and obviously to commercial exploitation if they really existed as popularly conceived. If you stipulate that all of those alleged psychic phenomena are inherently not exploitable in those ways or any other significantly profitable way, you're basically redefining those terms contrary to their popular meanings and just moving the goalposts.
The same could be said of antibiotics. However, before Ehrlich & Pasteur, there were no large scale use or even specific awareness of the concept (i.e. some cultures had folk medicinal remedies that involved antibiotics but the mechanism and specific compounds were not understood).
So until the 1880s, there was no commercial exploitation of these compounds, yet they nevertheless existed either in actuality or potential. Once there was an understanding of the principles and the development of some related chemical and biochemical processing, this exploded and had a dramatic impact on human life.
If someone postulated in, say, 1785, the idea that perhaps there were specific compounds that could be used to cure infection, and these could be produced en masse, would it not be analogous to respond that since nobody is commercially exploiting (at the time), they clearly do not exist?
> If someone postulated in, say, 1785, the idea that perhaps there were specific compounds that could be used to cure infection, and these could be produced en masse, would it not be analogous to respond that since nobody is commercially exploiting (at the time), they clearly do not exist?
None of the ideas included in the xkcd comic are particularly new or obscure. Despite being pigeonholed as a certain kind of bullshit, they are still ideas that a lot of people have heard about and been hearing about for a very long time—more than enough time for someone to get around to commercially exploiting if that were in fact practical. Ghostbusters came out 37 years ago, and the stuff it made fun of was familiar enough to its audience.
However, in 1785 the principles of chemical engineering were largely unknown, precluding the development of a modern-style pharmaceutical industry. It did not, however, prevent widespread use of antibiotic treatments that we now understand the mechanism for and can refine or synthesize into more effective forms.
(1) All of these practices depend on cultivating virtues that are contradictory to industrialization/capitalization: just as lyric poetry is.
(2) It is intriguing to consider commercial music as an industrial utilization of lyric poetry, but the point is that the individual activity of lyric poetry is not amenable to industrialization IN ITSELF, but it still exists. But perhaps this example doesn't work -- the really important argument is above, namely, that the cultivation of these practices -- astrology, dousing, etc, is individuated, and oriented around virtues directly opposed to their mass utilization.
(3) As other commentators have noticed, we also don't know that these sciences aren't used on a mass technological/industrial fashion --- it seems likely, for instance, that there are corporate entities that use i ching divination, and likely, too, that there are financial firms that utilize astrological methods -- but wouldn't reveal this because of the likely opprobrium from a still dominant (if clearly declining) orthodox materialism...
That’s because poetry’s success is a function of social consensus rather than objective reality. But anything that depends on something real can be industrialized.
If you don't have a theory of quantum mechanics and you don't have any observable phenomena that require a quantum explanation, then there's nothing to apply this kind of analysis to, and nothing to reject the existence of.
It’s a sound point. Even a very small edge in precognition or mind reading (no matter how unreliable) would allow hedge funds to beat the market and make a killing. Ergo these effects do not exist because nobody is doing this.
> Even a very small edge in precognition or mind reading (no matter how unreliable) would allow hedge funds to beat the market and make a killing. Ergo these effects do not exist because nobody is doing this.
It is known that many people who consider that they might have psychic powers are on the highly sensitive side.
Knowing people who are highly sensitive, I can easily imagine that the enormous stress, pressure and greed in a hedge fund environment would make hedge funds a horrible work environment for such people.
A single psychic hedge fund manager wouldn't reveal themselves, but as an industry if you're trying to hire psychics it would not be possible to keep this fact secret.
If you fail too hard at investing you have to quit, but if you’re too successful you also quit because you retire or the technique doesn’t work for too large a fund.
Note, “just hold index funds” as a strategy should be dominated by “borrow a lot of money and put it in index funds” by the same arguments. Don’t see robos offering that though.
What about his track record cannot be adequately explained by mundane insider trading?
Edit: Puzzled by the downvotes. I think it's entirely fair to ask whether Cohen is enough of an outlier that it requires a psychic explanation, even after accounting for the degree of success that can be reasonably attributed to luck and non-psychic skill (effects which other hedge fund managers are also subject to) plus the insider trading he's been involved in (which other hedge fund managers are not necessarily subject to).
In a world with a finite number of hedge fund managers, there is always going to be somebody who is the most successful out of the bunch, and that somebody is almost certainly going to appear to be an outlier even without a supernatural influence on their success. So before attributing anything to psychic powers, we first have to establish whether he's too successful to be accounted for by the non-psychic explanations. I genuinely don't know enough about Steve Cohen's track record to know whether he's that much of an outlier, and I'd appreciate some real information about the degree of his success rather than just downvoting.
Astrology, Tarot, and many similar things are absolutely used by people in financial planning, yeah. And on the more "acceptable" and orders of magnitude more people side of things, just look at "lucky x" with lottery purchases.
But that's still a LOT less profit exploitation than e.g. GPS which is ubiquitous. Curses haven't replaced drones in the military. Crystals aren't in billions of electronic devices wait
There would be evidence. The chart helpfully points out a specific kind of evidence that is more or less impossible to conceal with any kind of conspiracy.
A flipside example: Financial world could instead be used by individuals on casinos, betting, stock markets, etc. and they'd have incentive to keep it secret so they keep winning.
It might be plausible to assume that someone exploiting ESP or whatever to beat the house could keep the secret of how they're winning. But it's a lot less plausible to postulate that they could keep secret the fact that they are beating the house or beating the market, unless the effect size is too small to be of much interest in the first place (ie. you have to largely abandon part of the original hypothesis: that the crazy phenomenon actually works).
This argument has the structure of a conspiracy theory: "A small group of people being able to keep an enormous secret, in order to manipulate the rest of us."
The more enormous the secret and the larger the group of people who supposedly are in on it, the more the probability of the secret not being a secret very long approaches one.
I find it very unlikely that a shortcut to making a lot of money would be secret for very long.
In particular, as the scope of the conspiracy grows, the cost of keeping that secret very quickly outstrips any potential profit from exploiting the secret. Eg. if oil companies could use ESP to know where to drill, then they would still need to be spending large sums of money on computers and software to analyze seismic data, or else the collapse of demand for those products would expose the conspiracy.
no planned obsolescence took place for a number of consumer products.
multiple and costly military invasions were to offer democracy as goodwill and for world safety.
a certain vaccine I can't dare to name is a vaccine as it does provide immunity.
the institutions of most democracies surely can't be so corrupted.
worldwide cocoa isn't mostly harvested by enslaved workers.
HIV is so far more prevalent in Africa because they can't afford treatment and understand prevention measures there.
JFK.
I could go on and on but not sure how many examples you would need to accept that the chances for enormous secrets known by even a significantly large group to not take a freaking long time before blowing up is rather close to zero. thus reconsidering your opinion on the existence of conspiracies.
and, about shortcuts to making a lot of money being kept rather well secrets: dark budgets, secret and hidden inflation, supply fudging
A critique of conspiracy theories is not a denial that conspiracies exist. It just means that the concept of a "conspiracy theory" as basis of trying to understand reality is flawed, because it relies on no or bad evidence, and very often seeks to simplify a very complex reality in a way that doesn't do that reality justice.
But I am not sure if we both mean the same things when we use the word "conspiracy". It's not a secret, for example. that cocolate has a slave labor problem. Planned obsolescence is not a secret (but it's more complex than just someone trying to screw you over). I don't think I understand the conspiracy aspect of the rest of your examples, save for the obvious JFK conspiracy theories.
> about shortcuts to making a lot of money being kept rather well secrets: dark budgets, secret and hidden inflation, supply fudging
I would put to you: If the chances of this blowing up is rather close to zero, how do you know about it ?
definist fallacy or persuasive definition, not sure which argumentative trickery you are using right there with your selected definition of conspiracy theory critique.
what is more straight to see is that you've just made a false attribution by distorting what I wrote about close to zero chances of a blow up. I don't question your intention and honesty, but read my comment more carefully, or quote verbatim without distortion, and resist the use of fallacies in general if you would like to engage in honest and fair argumentative discution.
planned obsolescence was a well kept from public industrial scheme, decided by a small group, screwing billions of consumers. it isn't a secret anymore, it blew up, it took several decades to blow up, kept being refuted by some who haven't caught up with the evidences.
and if you accept the JFK case then I suppose you are admitting to beleive in some conspiracies, while at the same time refuting the concept while it's enduring its long process of battle against deniers, during the inevitable period of evidences remaining arguably to few or too weak, simplifications of reality.
consider at least accepting there is some contradiction this logic.
edit: had pressed send before finishing the comment.
I am seing one empty box that should get a "has been tried" mark and four that should get "are being used" marks. I don't know if they work but I know they are being used for profit.
This post really makes you think and challenged my world view when I first read it.
A while back I looked at the CIA "Star Gate" files[1], which, are a collection of documents released under FOIA detailing the CIA's investigation into psychic powers and whether or not those powers serve an intelligence gathering purpose. I just opened up random files and skimmed them - I wish I had taken notes when doing so, but my impression was that the CIA was actually producing some better than chance results with their experiments.
I don't really know what to make of parapsychology or psychic powers in general. Sometimes I'll idly wonder if I should dedicate time to trying to uncover my psychic powers.
Well this certainly makes me glad that the only time I was having to do statistical studies for research purposes, it was in the context of generated code performance.
Note the implicit (and, to most, justified) a priori convinction that psychic phenomena cannot exist, even if they have evidence to the contrary with similar confidence as other ("coventional") study subjects.
Isn't this an "isolated demand for rigor"? [1]
Doesn't this prove that science as practiced is not just the empirical process and some raw axioms that guide it (which these "psychic research" both follows), which scientists pretend to ascribe to, but also part blind faith/prejudice?
I'm not saying physic phenomena are real. I'm saying if tests are conducted with the same principles as those about physical phenomena, and results are equally carefully collected, and they have the same condidence, science in principle shouldn't treat them any differently than results about "regular" subjects.
>science in principle shouldn't treat them any differently than results about "regular" subjects.
That's the whole problem, as I understand it. If we are going to treat ESP research the same as research in other fields, we either accept ESP as real, or reject both ESP and other fields as deeply flawed.
Personally, I'm leaning heavily toward the second option. The methods in ESP are identical to methods in psychology, social sciences, etc, but not physics or chemistry. I'm just wondering where to draw the line. Maybe the whole method of statistical inference needs to be flushed? Maybe it cannot be saved by preregistration, publishing all the data, transparency and other bells and whistles.
>That's the whole problem, as I understand it. If we are going to treat ESP research the same as research in other fields, we either accept ESP as real, or reject both ESP and other fields as deeply flawed.
That's my point them. "If we are going to treat ESP research the same as research in other fields", according to the principles science pays lip service to this shouldn't be a question to begin with, because science shouldn't have a priori ideas about what can be treated "the same as other fields" and what shouldn't.
Research (results) should determine that (which means: if there are results in its favor, and we can't see a problem with the research that other - non ESP - accepted research doesn't have, we should accept it, as much as we accept in other such - non ESP - results).
Extraordinary claims require extraordinary evidence. (This isn't just some folk saying: it's a direct consequence of Bayes' theorem.) It's right to be more skeptical of a study that finds a result that doesn't have any known mechanism, doesn't match up with the rest of what we know, and isn't reproducible.
That's an a priori assumption though. Who arrived at the conclusion it was "extraordinary" and with what scientific methods? Not based on research, for one, because that's what we're evaluating here: research results positive to that claim. We could say: "We have a bias against it being true", but the "extraordinary claim" part is BS within the context of the scientific method.
As for applying Bayes, the "prior" (in fields where it's not mathematically determined, like e.g. judging whether a coin is fair), is just another name for "at will bias".
In this case, it was reproducible, though. And we'd have to reject many things in the history of physics if the criteria were "doesn't have any known mechanism"!
Does it matter? Sheldrake makes some wild claims, but eloquently packs them in a talk that is entertaining and at times thought-provoking. In any case, his braveness is certainly admirable. Worth a watch, I would say (perhaps with a pint of beer).
“ Parapsychologists are able to produce experimental evidence for psychic phenomena about as easily as normal scientists are able to produce such evidence for normal, non-psychic phenomena.”
Therefore psychic phenomena exist? I don’t see the problem.
It’s only a problem if you have strong priors telling you that they don’t exist, because that indicates a lot of other scientific fields are in trouble.
No, it's reasonable to phrase it as a belief that the phenomenon doesn't exist. That's a positive belief of the negative. The alternative of lack of belief in existence is a more neutral stance.
Skepticism doesn't mean resisting any justification for moving the needle of your certainty away from 50%.
You are allowed to expect that psychic phenomena being real would lead to the existence of clear evidence that psychic phenomena are real, and to conclude from the lack of such evidence where it ought to be found that it is less likely that psychic phenomena are real.
> Suppose you have a target that is half red and half blue; you are aiming for red. You would have to be very very confident in your dart skills to say there is only a one in a million chance you will miss it. But if there is a target that is 999,999 millionths red, and 1 millionth blue, then you do not have to be at all good at darts to say confidently that there is only a one in a million chance you will miss the red area.
> Suppose a Christian says “Jesus might be God. And he might not be God. 50-50 chance. So you would have to be incredibly overconfident to say you’re sure he isn’t.” The atheist might respond “The target is full of all of these zillions of hypotheses – Jesus is God, Allah is God, Ahura Mazda is God, Vishnu is God, a random guy we’ve never heard of is God. You are taking a tiny tiny submillimeter-sized fraction of a huge blue target, painting it red, and saying that because there are two regions of the target, a blue region and a red region, you have equal chance of hitting either.” Eliezer Yudkowsky calls this “privileging the hypothesis”. [0]
> There’s a tougher case. Suppose the Christian says “Okay, I’m not sure about Jesus. But either there is a Hell, or there isn’t. Fifty fifty. Right?”
> I think the argument against this is that there are way more ways for there not to be Hell than there are for there to be Hell. If you take a bunch of atoms and shake them up, they usually end up as not-Hell, in much the same way as the creationists’ fabled tornado-going-through-a-junkyard usually ends up as not-a-Boeing-747. For there to be Hell you have to have some kind of mechanism for judging good vs. evil – which is a small part of the space of all mechanisms, let alone the space of all things – some mechanism for diverting the souls of the evil to a specific place, which same, some mechanism for punishing them – again same – et cetera. Most universes won’t have Hell unless you go through a lot of work to put one there. Therefore, Hell existing is only a very tiny part of the target. Making this argument correctly would require an in-depth explanation of formalizations of Occam’s Razor, which is outside the scope of this essay but which you can find on the LW Sequences.
this is similar to a kind of subject and object confusion:
subject: most reasonable people
object: psi phenomena
verbs: to believe, to exist
The full statement can be stated in at least two ways:
"[ these people ] don't believe that [ these phenomena ] exist."
"[ these people ] believe that [ these phenomena ] don't exist."
Either one is valid, as are the contracted versions, but the first one describes the absence of belief in existence, and the second describes the presence of a belief in non-existence.
These are closely related things, but not identical, and I imagine the author likely chose carefully.
In the years that I've read him I noticed this is something that happens from time to time with Scott Siskinds (see 3 if you take offense to me using his name) texts, especially when he steps out of his zone of competence to opine on politics, science in general, AI etc. The simple, obvious answer that the establishment believes in is met with some snarky ridicule and on goes the article, with some very subtle rhetoric veiled in the irony ("rigour doesn't matter, all scientists have biases, you can't trust rigorous science if the wrong people do it" is the message behind [quote] Studies are going to be confounded by the allegiance of the researcher. When researchers who don’t believe something discover it, that’s when it’s worth looking into. [/quote] ).
I often wondered why this is, because he seems like a very smart guy with interesting ideas sometimes, so it seemed kind of weird and conflicting.
Then I learned he's trying to hide his beliefs about scientific racism and political views (1) while often writing in the tone of "pained centrist just wanting to understand things scientifically" and it made much more sense as (in the worst case) an old school propaganda technique of FUD in science and rigorous science to make the HBD go down better and (more charitably) as Mr. Siskind just being another bright-but-human person who when writing on things he has actual expertise on can be very well worth reading, but simply shouldn't be trusted on anything aside from this. Rationalwiki has a decent writeup on him (2)
I thought it was a fascinating write up about how if you assume ESP doesn't exist then parapsychology research has something important to say about the replication crisis in social sciences.
Going into how double blind solves all this means skipping the interesting bit: that parapsychology can be viewed as a control group for the scientific method.
Exactly. He's definitely smart, and so it was a mystery to me why his articles were so incredibly long and rambling. (I get that some people enjoy his prose regardless of content, but I'm not one of them.) Why would a smart person who in theory wants to convey important points be so obscurantist about them? I threw up my hands and put it down to a personality quirk.
Unfortunately, while the Sandifer article does make some valid points about Scott Alexander's writing, it also deploys plenty of questionable rhetorical tricks and shoddy reasoning itself. I'm personally not impressed by either article.
Well, a quick red flag in Sandifer's article is that she describes Slate Star Codex as "a crackpot blog that keeps giving cover to f---ing nazis".
But what first got my attention (well, second, since the first thing was that Sandifer's article is as long and rambling as Alexander's is--both of them could have made what valid points they had with far fewer words) was this statement, a little earlier, about Scott Aaronson (referring to the comment in the blog post of Aaronson's that Alexander's article links to):
> Aaronson was declaring that, as a shy and nerdy man, he was a member of "one of society's least privileged classes" in response to someone saying they were a rape victim.
The reason this got my attention was that I had previously read the Aaronson post in question, and the comments, and I didn't remember any such thing. So I read through the post and comments again. The commenter that Aaronson was responding to when he talked about his personal history (the comment of his that Alexander quoted) did say she was a rape victim in that comment thread--but she said it after the quoted comment from Aaronson. Aaronson's quoted comment was in response to an earlier comment of hers, which did not say anything about her having been raped.
So Sandifer lied in her article about a fact that was key to her argument. I understand this is just one example, but there isn't room here for a multi-thousand-word critique and I don't have time to write one anyway. But the example I have given is not the only questionable thing I saw.
What's your evidence that there was a lie, as opposed to just a reasonable error of fact or an editing error?
Amy did imply she was a victim of sexual assault previously, just not rape. So at the very least it was an error of degree. And regardless, Aaronson, had no reason to think he wasn't talking to a rape victim. The odds were something he surely knew. So I think Sandifer's correct that "this was neither the time nor the place to express them". And that's coming from a "shy and nerdy" guy.
Given that you're very up in arms about "questionable rhetorical tricks and shoddy reasoning", you'll see why I would expect you to do better than turning a reasonable and non-fatal error into a supposed lie about a key fact. But it is about what I'd expect from somebody working hard to defend a cryptofascist.
That's definitely not true. I just agreed with you that there was an error in Sandifer's article. We agree that questionable rhetorical tricks and shoddy reasoning are bad.
But if you can't talk to me or take responsibility for your own words because I have a negative opinion of one of your faves, that's a choice you're making.
And I really object to the comment I'm replying to getting flagged. I get that some people will disagree, but flagging it as if it were somehow inappropriate is a misuse of flagging power.
Depends what you and him exactly mean, but since he's talking about the neoreactionaries being correct, I'll assume he doesn't mean the scientific version which posits that race has no genetic basis (ctrl+F "race")
Even if it was correct, I think if there are no nefarious consequences being pushed for, I think people who want to argue HBD should rise to the challenge of explaining in plain english what they mean and what consequences they would draw from their assumption being correct, instead of hiding behind obfuscation. While I am not necessarily 100% aligned with groups like AntiFa chapters, BLM etc., at least they are honest about what they think and want to achieve.
For hundreds of years people have energetically misused science to justify both their racist feelings and systemic racism. And other people have showed up to disingenuously defend that in many ways, including your evasive, just-asking-questions routine.
There is no reason to think that activity has stopped. And indeed, the shift in the last 50 years to make overt racism moderately less socially acceptable suggests that we might see an increase in the amount of covert racism. So I feel no obligation to take the latest iteration any more seriously than I would the 1850s version. It's the same deal with perpetual motion machines or multi-level marketing schemes or religious door-knockers: the next one to come along could finally be the one that isn't horseshit. But that's not the way to bet.
If you'd like to make the argument that your personal favorite scientific racism is better science than the scientific racism favored by other racists, feel free to take a swing at it. But just looking over your comment history, I'm not going to give it a lot of weight.
Just evaluate evidence and methodology. Historical factors are irrelevant. You obviously have a preconceived Truth that could never be disproven no matter what was empirically shown, but that's not science.
Nah. I don't have The Truth, just some pretty good heuristics on how I spend my time. I've all ready looked into enough religion to know that I don't have to give any time to godbotherers. I've already looked at enough perpetual motion machines to know that I don't have to disprove each one. And I don't have to hear out each racist on why this time he's rilly rilly rite about how the people he hated all along are scientifically bad.
Is it possible that one day somebody will come along with a real perpetual motion machine and I'll miss out for a while? Sure. But there are other people who enjoy evaluating that stuff enough that I'll eventually hear about it. And it's the same deal here. I don't get my theology from evangelical Christians, and I don't get my bioanthropology from evangelical racists.
Finally, they would conduct the experiment in a series of different batches. Half the batches (randomly assigned) would be conducted by Dr. Schlitz, the other half by Dr. Wiseman. Because the two authors had very carefully standardized the setting, apparatus and procedure beforehand, “conducted by” pretty much just meant greeting the participants, giving the experimental instructions, and doing the staring.
The results? Schlitz’s trials found strong evidence of psychic powers, Wiseman’s trials found no evidence whatsoever.
Take a second to reflect on how this makes no sense. Two experimenters in the same laboratory, using the same apparatus, having no contact with the subjects except to introduce themselves and flip a few switches – and whether one or the other was there that day completely altered the result. For a good time, watch the gymnastics they have to do to in the paper to make this sound sufficiently sensical to even get published. This is the only journal article I’ve ever read where, in the part of the Discussion section where you’re supposed to propose possible reasons for your findings, both authors suggest maybe their co-author hacked into the computer and altered the results.
Well, if you would take the "psi can never exist" blinders off, Mr. Codex, then this seems to be a pointer towards "this works better if you believe it will work", and it makes perfect sense if you accept this.
Schlitz's belief that it would work resulted in it working. Wiseman's belief that it wouldn't resulted in it not working.
The whole idea behind psi is that the human mind can sense and/or effect the world in currently-inexplicable ways, and here it is doing just that. With at least ten other experiments on this idea getting similar results. But parapsychology is completely and utterly a scam in your eyes so that is not a result you will consider.
One can get quite a bit of popular traction from arranging for one's theory to be effectively-impossible to disprove (e.g. "it only works if you believe in it during the experiment"). Not much explanatory power though.
The experiment I excerpted his description of seems to go along with exactly that. Experimenter who believes? Things happen. Experimenter who does not believe? Nothing happens.
https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-...
Rather than publish all of the dubious speculations and the analysis whose problems took time to emerge, simply publish the data. The advantage, in his words:
> In the example of the ESP study, if anything’s valuable it’s the data. Publishing the data would get the journal off the hook regarding fairness, open-mindedness, and not missing a scoop, while enabling others to move on reanalyses right away, and without saddling the journal with an embarrassing endorsement of a weak theory that, it turns out, was not really supported by data at all.