The real issue is that experiments that give "expected" results are not subject to this kind of scrutiny. Thus experiments are much less trustworthy than one would assume. It sometimes takes decades for errors in experiments to come out - eg. from "Surely your joking, Mr. Feynman":
Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of - this history - because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong - and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...
Actually, most major experiments in particle physics these days (including the OPERA experiment) avoid this sort of confirmation bias by being run "blind." The scientists write all of their data reduction pipelines before taking any actual data and test their pipelines on simulated data. When they are confident that their pipeline is running as expected they run the experiment, put the data through their pipeline and publish the result, no matter how unexpected it is.
As the OPERA result showed, it has the problem that if you don't understand everything in your experiment perfectly (which is difficult to do in a very large, complicated experiment) you run the risk of embarrassing yourself by making some obvious-in-retrospect mistake and publishing an obviously absurd result. But in the long run it's not so bad a price to pay to avoid the sort of confirmation bias that Feynman was talking about.
Physics is far ahead of other disciplines in this regard. Choosing your statistical test after you gather the data, selectively removing "outliers" after you gather the data, non-blind interpretation of pictures by humans who have a stake in the outcome and only publishing statistically significant results are all par for the course in e.g. neuroscience.
This paper [1] points out that commonly-used measures of statistical significance are downright meaningless when additional degrees of freedom are hidden in the way you describe.
It's even worse than that paper describes and this is something that every statistics 101 class worth its salt points out: if you are allowed to choose a statistical test after you've gathered your data you can prove any conclusion you want with arbitrarily high confidence. Note that the paper does not list choosing a test before you gather the data as a requirement. The only way to do meaningful statistics is the way splat describes: describe exactly how you're going to analyze it before you gather the data, then send the paper to a journal which decides whether to publish it before the data has been gathered, and then complete the paper by actually doing the experiment and adding the data to the paper.
>The scientists write all of their data reduction pipelines before taking any actual data and test their pipelines on simulated data. When they are confident that their pipeline is running as expected they run the experiment, put the data through their pipeline and publish the result, no matter how unexpected it is.
How does that help when the prediction matches the measurement but the experiment is flawed ?
What you describe is more or less what Lakatos coined the "Research Programme" after. [1]
In a nutshell, we know that science more or less progresses from one theory to another after the existing theory has been falsified (Popper), but this transition does not happen overnight, even after a falsifying result has been put forth. That is because science is really structured with a "hard core" hypothesis (say, in physics, that nothing moves faster than c), and various "auxiliary hypotheses" that bear the brunt of criticism before the core belief is attacked (say those that posit the integrity of the measurement instrument, for example). And that is all that is going on here. There are many, many theories that are going to be dismantled before physics transitions to something post-relativity. It's just the way the assumptions are structured.
While this may seem trivial, we see this playing out in the battle of classical against behavioral economics. Are the results of experimental psychology admissible in the criticism of certain economic models? Models don't claim to be perfect, after all...
Surely this is just a reflection of the fact that science is performed by humans with limited resources. I mean, there is something to be said for getting the approximately result correctly many times over.
And don't forget there is a check and balance in the form of meta-studies and statistics that are sometimes uncanny can exposing bias and flaws.
Finally - this highlights the dual nature of 'experimental' vs 'theoretical' physics, and they work so well in tandem.
Agreed, its the nature of science. That's why theory is so important and why physicists pay as much importance to non-experimental features like "beauty" when judging theories. Experiments cannot be trusted unless explained by theory!
To me the real lesson of the Millikan experiment is that science moves slower than you think. It does take decades, sometimes, for people to notice the tiny but consistent discrepancy between their expectation and the data, but that's not necessarily a sign of failure; that's just the time scale of success.
When Millikan's experiment was new it was new; people hadn't done it before, there was slop to work out, and the theory was miles behind. These things take time to converge on a consensus. Sometimes a long time. Also, money.
The lesson isn't that Millikan screwed up, everyone can screw up at some point. The lesson is that folks had a chance to show Millikan was wrong simply by rerunning his experiment and coming up with the correct result. But they didn't. They didn't have the integrity to challenge Millikan's result. They turned every dial they could to fudge the numbers enough to end up with a result near Millikan's.
How would such dynamics pan out for experiments that aren't amenable to the sort of polite gradual adjustment as Millikan's was?
The real lesson is that this sort of thing (fudging numbers, lack of scientific integrity) is quite prevalent but most of the time goes unnoticed because there isn't something as dramatic as with the Millikan experiment to show it plainly.
A similar case is Hubble's measurement of the expanding Universe. The Universe is expanding, and Hubble's linear velocity = Hubble_constant*distance law is an excellent first approximation, but:
1. Hubble was actually measuring the local motion of nearby galaxies, the so called Local Group. These galaxies are too close for Hubble expansion to matter, they're just gravitating.
2. The actual value for the Hubble constant he measured was way off, by something like 10x.
1. Hubble's draftsman made an error while preparing his famous diagram. From Figure 1 of his paper (http://adsabs.harvard.edu/abs/1931ApJ....74...43H), you'll notice that the vertical axis is in "km" rather than "km/sec."
2. Measurements of the Hubble Constant over time: http://www.pnas.org/content/101/1/8/F2.large.jpg For some reason it's not plotted logarithmically so it's hard to see the convergence towards ~70 km/s/Mpc in the last twenty years.
Interestingly, there was a long fight in the field of cosmology from ~1960 until ~1990 as to whether the Hubble constant was 50 or 100. It wasn't until within the past twenty years or so that the community began to agree upon the now-accepted value of ~70 km/s/Mpc.
> The cables house 36 strands of superconducting wire, each strand being exactly 0.825 mm in diameter. Each strand houses 6300 superconducting filaments of Niobium-titanium (NbTi). Each filament is about 0.006 mm thick, i.e. 10 times thinner than a normal human hair.
> tolerances are only a few micrometers.
> Total superconducting cable required 1200 tonnes which translates to around 7600 km of cable (the cable is made up of strands which is made of filaments, total length of filaments is astronomical - 5 times to the sun and back with enough left over for a few trips to the moon).
I'm really impressed with how open this group was to be proven wrong and how hard they worked to rule out or confirm error on their part.
Compared to Wolfe-Simon et al (the group that claims to have found arsenic in the DNA of certain bacteria) they show exactly how science should be done.
Standard caveat: while Science is an extremely reputable outlet, this blog post is unsourced. There hasn't been an official statement from the OPERA collaboration yet.
The best comment on this, courtesy of Ars Technica:
>At the AAAS meeting's discussion, CERN's director of research, Sergio Bertolucci, placed his bet on what the results would be: "I have difficulty to believe it, because nothing in Italy arrives ahead of time."
"The OPERA Collaboration, by continuing its campaign of verifications on the neutrino velocity measurement, has identified two issues that could significantly affect the reported result. The first one is linked to the oscillator used to produce the events time-stamps in between the GPS synchronizations. The second point is related to the connection of the optical fiber bringing the external GPS signal to the OPERA master clock.
These two issues can modify the neutrino time of flight in opposite directions."
The best pessimistic response I read to this experiment when it first came out stated that overturning well-established theorems in physics should be left to experiments where the result is clearly and simply binary, not where the result is an extraordinarily precise measurement in a complex system.
Otoh, one of the early tests for general relativity was a minor deviation in Mercury's orbit from what was predicted by classical mechanics. My rough intuition is that inconsistencies can show up initially as faint flickers, but once you investigate more closely, one can find interesting and prominent counter examples.
I'm not an expert on this, but I believe the deviation in Mercury's orbit was easily observed and calculated, and thus became in a sense a binary call-out of Newtonian physics. Until highly precise measurements of relativistic velocities is common place, experiments like this are of course valuable and interesting, but one instance of an experimental result like this cannot be relied upon to supersede special relativity.
Anything which is going faster than C in one inertial frame of reference, is going backwards in time in another inertial frame of reference, which allows for a violation of causality. I'm sure the experimental set-up would be complicated as balls, but a qualitative property like causality is probably more robust to measurement.
One small disagreement with you. While general relativity suggests that going faster than C causes a violation of causality, it might be the case that things can go faster than C without causing violation of causality, and we need a new model for such cases.
Part of the point of experimentation is to find holes in our existing models, so it is reasonable for scientists to look for faster than C without violation of causality.
It does not necessarily allow for a violation of causality.
As an example, suppose that we were able to go faster than C in the reference frame of the fixed distant stars, but not in other reference frames. There are reference frames where you can go backwards in time, but none in which you can violate causality.
(That said, General Relativity allows for causality violations. However setting them up is well beyond any engineering ability our species is likely to develop...)
the description of the error appears just too vague in my ears. due to "bad connection between a fiber optic cable that connects to the GPS receiver" seems unlikely, since it's first of all a very systematic error and secondly I can't fathom how a fiber optic cable can have a bad connection - at least not in this setting... unless they would've unintentionally bent the cable, resulting in a higher error rate in the transmission of the signal but shouldn't that show up in some network diagnostic tool?!
If the connector was bad or broken, there could be an air gap between the fiber and the diode, creating an unwanted refraction that could introduce transmission error.
I thought they already proved that the faster-than-light neutrino was due to relativity between the GPS satellite, the origin, and the destination? Did I just hallucinate that?
This comment got me wondering how important it is for navigation to take into account relativistic time dilation in GPS. Apparently: very. Turns out that ignoring it would cause a drift in position of about 10km/day. http://www.astronomy.ohio-state.edu/~pogge/Ast162/Unit5/gps....
From what I read, that was just a suggestion from someone's blog that showed up on HN and reddit as if it was an accepted fact due to a poorly worded headline.
I didn't downvote, but nathnb's comment isn't adding anything to the discussion. It's a personal aside about some confusion he is having; his last question especially emphasizes that aspect.
But I see that several other comments at that level were also downvoted.
What exactly do you think they were doing for the six months between finding their results and publishing them? Are you under the impression that the OPERA scientists just randomly dump unvetted data on the public?
You labor under a false assumption of what the process of science is or should be.
Science is not a process of walking from one stepping stone of absolute, solid truth to another stepping stone of truth. Science is messy. Because the world is messy. Science is the process of observing the Universe as best we can given our limits and attempting to verify various models and conjectures about the way the Universe works. But along the way there are many potentials for pitfalls. Every experiment has a degree of error due to factors unrelated to what the experiment is trying to measure.
Published scientific papers are not like chapters in a text-book, they are merely the results of experiments. Sometimes it's possible to have experiments which seem to support one result or another but turn out to have some flaw or merely involve factors which we cannot explain yet. For example, we still don't know what "dark matter" actually is even though we have a pretty good idea that it does exist and some rough estimate of some of its properties. And for decades we did not know the origin of gamma ray bursts. It's the rare experiment that is definitive enough to provide unambiguous support or refutation for a specific theory, most experiments are somewhere in the middle, somewhat muddled, imperfect, and generally only gain strength once independently repeated and producing the same result.
They didn't claim to have proved Einstein wrong. To continue the weird analogy: They looked for the bug themselves, and couldn't find it, and then asked other people to find the bug.
pretty much. the mess newton caused by keeping calculus a proprietary technology for so long (due partially to existing practice in the alchemy industry and partially to being completely nuts) made it pretty clear that some degree of openness was necessary to really get anything done.
And they presented this openly for it to be checked. Einstein was not proven wrong because of this and Physics textbooks were not rewritten because of it. This was a great was to handle something that seemed very suspicious and trying to find out if it was correct. They did everything on their end and then presented it to the public for further scrutiny so I am not sure what you wanted them to do.
In addition to the other comments, it should be noted that this happens after the fact. You don't really know what your bugs will do until they turn up, which you can only know after you run the program.
Similarly, they didn't know what their results would be until after they ran the experiment. At that point, they can either: share the results, or hide the results. Hiding the results is the absolute worst thing that any scientist can do. The only time the option not to share the results would be acceptable is when they can be discredited or discounted (and sometimes they should still be shared). Since they couldn't discredit their results (they tried), they took the only responsible option remaining.
Indeed, they didn't just publish the results, they asked others to look for the bug. Like others have said, this is the best possible outcome.
The point was - you can't check everything beforehand. No matter how hard you try, there's always a non-zero probability that a bug will "prove Einstein wrong", or drive people into a ditch, or find horribly mismatched partners on a dating site, or explode toasters and kill puppies.
Actually, from the beginning they've mentioned that equipment failure would be one of the ways to account for the findings. They also have been incredibly open in asking for help to explain the anomaly. I don't see this as scary at all. Science has come out looking incredibly even keeled and reasonable. Some of the news outlets that have covered the science... less so. But still, almost every article I read had all the caveats listed even if there were have some overly hopeful headlines.
Oh, "scary" is the right word, notwithstanding everything you just said, which is all true.
It takes courage to stand in front of every camera on earth and claim to have measured something impossible, no matter how cautious you are, and no matter how gracious everyone else is. It takes courage to knowingly and deliberately turn yourself into the butt of jokes in foreign languages whose names you don't even know, all on the very slim chance that this thing you can't explain is something breathtakingly awesome, Isaac Newton-awesome, Albert Einstein's 1905-awesome. It was bound to be embarrassing in the end, and lo and behold it is shaping up to be exactly as embarrassing as every one of my fellow experimentalists knew it would be, and I can't decide whether to laugh, cry, or salute, because when you've spent months or years of your life in utter despair, trying to get your experiment to produce something halfway believable, or redoing six months of work because a broken fridge probably contaminated the first batch, or trembling as you cross-check the simulation code the week before your thesis is due, you've learned how it feels: Awful.
I'll go with "salute": Let's all raise a glass to these folks and be grateful that they are on the road to finding their problem, rather than being haunted by uncertainty forever. May their next result be twice as exciting and only half as wrong!
It has little to do with science, and I don't understand your comment about science being scary. Science is simply method used to discover how things actually work. There is nothing "scary" about that.
This is more a question of engineering. Doing modern physics experiments require extremely complicated machines with tolerances so fine that the tiniest amount of noise in the results can throw off the accuracy of the measurements.
The experimenters were not some quacks making spurious claims. They very, very, very carefully checked and rechecked their results before deciding to talk to the outside scientific community and asking for help.
Given the complexity of these machines, it doesn't surprise me at all that a single cable might be the source of the errors. If you write code, you find many times throughout your career you get errant results. In the process of tracking down the cause, you may spend hours, days, or longer going line by line and missing the problem. In the end, it may turn out to be a mis-named variable or the wrong kind of comparator (== vs === in js). Simple things that are easy to miss.
There was a story recently about a commonly used algorithm that dates back decades that had a bug in it. When the algorithm was originally developed, it was never thought that it would still be in use 30 years later and disregarded the size of data sets available. When the algo was used on a very large, modern set of data an integer would flip and go negative. The point is, its easy to miss things in very complicated systems and very hard to make sure there are little or no bugs at all. I don't remember what type of algo it was, but it was an interesting read.
You're probably referring to is, "Extra, Extra - Read All About It: Nearly All Binary Searches and Mergesorts are Broken", discussed here: http://news.ycombinator.com/item?id=1130463
Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of - this history - because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong - and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...