>"if random noise is behind the patterns, says Afshordi, then the chance of seeing such echoes is about 1 in 270, or 2.9 sigma. To be sure that they are not noise, such echoes will have to be spotted in future black-hole mergers."
We are getting closer and closer... So finally we see a correct interpretation of a p-value in the media, but the connection to the following sentence is not clear so I am not sure the meaning was really understood.
How does spotting more such echos allow us to "be sure they are not noise", and how does this relate to that 1/270 number?
If the probability of such an observation was 1/1.7 million assuming a random noise model (rather than 1/270), would that mean we could "be sure it was not noise"? Shouldn't that depend on how well the observations could be fit by alternative models?
General relativity makes certain predictions, we now measured a deviation from those predictions assuming a specific model of the source of the observation. So more measurements will help to build up confidence that the deviations are not just measurement errors.
After that you have to explain those deviations and the two most obvious options are that either the model of the event or the theory itself is wrong or at least not accurate enough to describe the observations. Then you can look for changes to the model, varying masses or the number of objects involved, or maybe even look at entirely different possible source of gravitational waves to explain the observations.
But if you return with empty hands you will have to take the option serious that a modification of the theory is required. For example if you theory only allows a quantity to decrease monotonically over time but your observations show oscillations, then you have pretty strong evidence that your theory requires modifications.
You throw a dice 3 times. It always show 6. How do you know the dice is loaded, and that it wasn't a fluke? Repeat the experiment, and see if it starts looking random (pattern disappears) or if the pattern is strengthened (always saying 6).
Also, I couldn't get the original documents at the time, but you reminded me of this:
>'The simplest assumption about dice as random-number generators is that each face is equally likely, and therefore the event “five or six” will occur with probability 1/3 and the number of successes out of 12 will be distributed according to the binomial distribution. When the data are compared to this “fair binomial” hypothesis using Pearson’s 2 test without any binning, Pearson found a p-value of 0.000016, or “the odds are 62,499 to 1 against such a system of deviations on a random selection.”'
https://galton.uchicago.edu/about/docs/2009/2009_dice_zac_la...
The point is you will always find deviations (with extremely low p-values) if you look hard enough. It is about collecting data as carefully as possible, and determining which model fits best, not which fits perfectly.
> The point is you will always find deviations (with extremely low p-values) if you look hard enough
I don't know how you can get that point from the article instead of 'an 1894-era die is biased but you need a lot of statistical power in order to see that'.
I think the lesson is clear... the more messed up your methods the easier to see the deviation (ie the historical vs modern experiment). Also, where do you get this:
>"you need a lot of statistical power in order to see that"
For a rational person, it depends who gave you the dice. It sounds like you just put a low prior on dice being loaded since you are so used to getting well made dice from neutral parties...
You can factor in the expected false positive base rate and the confidence to figure out by how much you should boost your prior expectation of the theory being correct. "Being sure" is shorthand for "having a prior very close to 1 or 0".
We are getting closer and closer... So finally we see a correct interpretation of a p-value in the media, but the connection to the following sentence is not clear so I am not sure the meaning was really understood.
How does spotting more such echos allow us to "be sure they are not noise", and how does this relate to that 1/270 number?
If the probability of such an observation was 1/1.7 million assuming a random noise model (rather than 1/270), would that mean we could "be sure it was not noise"? Shouldn't that depend on how well the observations could be fit by alternative models?