We've only observed a small handful of black hole collisions though so seeing something at 2.9 sigma would be somewhat more surprising, right? Definitely not rising to the level of a discovery yet of course.
Only a small handful of observations (only one, last I heard, but I may be out of date), but lots and lots of analyses. And for every scientist that has their own model and does their own data processing, there's a chance that the model lines up with some arbitrary noise in the data.
You also have to add in that once these experimental results were released that a hundred or so theoretical physicists immediately started working on massaging the data into supporting their pet theories. After every anomalous result there's immediately hundreds of published papers by someone trying to simply be the first to publish in case their idea happens to pan out. It is a kind of shotgun approach to winding up being the next Dirac discovering the positron from math or whatever.
Since there's 99 other PhDs who probably looked at this data and found their pet theories didn't match the data and haven't been able to publish you have to account for that filtering effect that this was the 1-in-100 paper that managed to match the data. Adding that "Look Elsewhere Effect" to the 2.9 sigma would push the global significance (in the literal sense of global -- meaning all the research teams across the whole world) of this result down into meaninglessness.
Of course its likely that any discovery would start out looking like something on the edge of significance exactly like this. The safe bet is that this disappears, but all we can do is wait for more data to come in and see if the significance improves or disappears.
And I do really hope that someone finds something like this via the LIGO data. I'm convinced there's something very interesting out there to find, and sooner or later it should pop up experimentally and shake up our model of the universe.
Yes, this is basically a distributed version of the issues discussed in psychology regarding researcher degrees of freedom with a single data set. If you throw enough models at the data, one of them is bound to stick, whether it is predictive or not.
