Is it true that 2σ holds some significance? I imagine if it was common to get erroneous 2σ or 3σ readings this wouldn't have made the news in the way that it did.
Just trying to determine if this is closer to "this is probably something but we will need to confirm" or "something weird happened but it's probably nothing"
Let's put it this way: the LHC has on the order 2000 results so far, assuming the null hypothesis, you expect about 100 >2σ deviations and about 5 that are >3σ.
The combined significance from ATLAS and CMS will be higher, but it's quite an involved process to get all the systematics right and get approval from both collaborations.
Although in "real" terms 2σ is still pretty significant - but not firm enough to go "yes, absolutely" - but if they've got 2σ it's unlikely to be just noise, albeit still perfectly possible. IIRC 2σ is about a 1/20 chance of being erroneous. 1σ is almost meaningless - 2/5 chance of being erroneous.
The significance of 2σ really depends on the context. If we are looking for something at a particular place in the energy spectrum and we see a 2σ detection in the right place . . . well, that is tantalizing but not convincing. However, in this case, we are looking for unexpected signals across a whole range of energies. By sheer random chance, there will be a number of 2σ peaks that are meaningless noise. It gets a little more interesting if we see weak detections in two different experiments because the combined likelihood of two experiments having random peaks at the same place is considerably smaller. However, 1.2σ is nothing but noise, so I would say nothing to see here . . . at least, not yet.
1/20? So if 20 tests were performed at the facility, we'd expect at least one of these to appear spuriously? Presumably there have been 20 tests performed there?
This failure mode is correct and very real. It's actually one of the reasons why we're starting to see significant pushback against p-values. https://xkcd.com/882/
ATLAS observes the peak with a global significance of 2σ http://cds.cern.ch/record/2114853
5σ is the gold standard for a discovery