More to the point, who cares? The storm was still pretty bad: this isn't a case of a "bust" forecast. Long Island got hit, as well as much of New England.
So the line on the map of where the heaviest snow would be was a bit off: the only reason this is "news" at all is because that line happened to affect New York City, and there are a lot of journalists who live there whose view of the world matches that New Yorker cover[1]. If the line had been anywhere else we wouldn't have heard about it at all.
Fair enough, but it's worth noting that line is the difference between empty ocean and twenty million people, threee major airports and the financial capital of the world. It's sort of the definition of a distinction denoting an actual difference.
I think the problem is that people don't really understand nonlinearities in these things.
Imagine somebody aims a gun at you and I say, "Watch out, he's going to shoot you!" Then he fires the gun, but it misses you by an inch. You wouldn't say "you idiot, the bullet didn't hit me at all." But the weather equivalent has people thinking forecasters are stupid.
Except that (probably) no one predicts where bullets go for a living whereas, for weather reporters (or, more accurately, meteorologists), it's their job.
I theorize if we started smashing them in the balls with a tackhammer every time they're wrong, you'd see both a mass exodus out of that field and a marked increase in accuracy.
Given the continual clickbait alarmist stuff about it being a snow-hurricane (say what?) I'm surprised the hurricane analogy hasn't stuck deeper. Technically hurricane Katrina never hit New Orleans, it hit Slidell. I can assure you no one cares, other than maybe people living in Slidell. The rest of the world feels apathy due to lack of geographic knowledge. Boston, NYC, Long Island, whatever. Similar to the Katrina example, I'm sure the difference between LI and NYC is extremely important to the status seeking locals, but...
Did you forget about the entire city of Philadelphia? 14 inches predicted, < 1 inch actual snowfall.
The city is practically a ghost town today thanks to a botched weather forecast. Tons of businesses and schools closed preemptively because they were predicting massive snowfall totals with what seemed like absolute certainty. According to meteorologists here, it wasn't about whether or not we were getting snow, it was about whether we were getting 1 foot or 2 feet.
Did you forget about the entire city of Philadelphia? 14 inches predicted, < 1 inch actual snowfall.
OK, but so what? This is about unpredictability, not over-estimation. If the unpredictability was better conveyed the best case scenario is still everyone preparing like they did for a storm that didn't hit. Worst case is suffering hardships due to underestimating the storm.
I'm a bit further northeast and it's coming down pretty hard outside my window right now. The preemptive cancellations really did avoid a lot of unsafe travel here. We have a bit more familiarity with the unpredictability of snowfall predictions, but it usually manifests as people saying: "It won't be that bad" and getting into trouble the few times it really is that bad.
It didn't hit the cities that aren't used to big snowfall. But those cities house a lot of national media, so now we get to listen their scorn for the next week.
I would rather they close and not get snow than to stay open and put people in danger. Last year in Georgia the weather forecast was so bad and officials so late reacting that thousands were trapped at work and tens of thousands spent hours just driving home.
There is a very significant economic impact during high-impact weather-related decision making. The city of NY is missing out on a huge amount of money due to the "poor planning" - additionally, so is the rest of the world if flights were cancelled that did not need to be cancelled. Decision making is important in NYC, and providing the best possible information to the decision makers is a critical aspect of modern meteorology.
Maybe you don't care that it missed NYC, but Delta cares. Minimum wage workers who couldn't get to work and missed a shift care. The Mayor cares. etc.
But the problem is a problem of planning for uncertainties. Let's say you know its 50/50 chance of a "bad storm" and you have to make a decision to flip the "emergency prep" switch. You now have four possibilities with associated costs. (and that's a drastically simplified analysis)
With the right data, you can come to some level of statistically optimized set of decisions, but that won't ever completely eliminate the instances when you turn on emergency actions, but the storm doesn't follow up. It merely minimizes them...
Really, if all these private and public entities care about the accuracy of forecasts, maybe they should have made more noise when NOAA programs were being gutted during the budget process and long term decisions were being made about satellite programs... I can't imagine what the handwringing will be if the satellite coverage gaps come to pass. Maybe we'd actually learn that certain investments in common needs actually pay off quite well.
Most people don't do any real work anyway (just fake, fluffy work that seem like work). So they didn't get paid. So it didn't get wasted. Economy impact positive, not negative.
While this article mentions that the GFS model was more accurate than the others (ECMWF & NAM), it neglects to give the backstory as to why it was more accurate and less trusted. For years the GFS has been made fun of because of its poor performance when compared to the Euro - and it just got the major upgrades it needed this month! New supercomputers, new high-res models, all went online just before the storm arrived. So, very few trusted the model even though it was in the best spot to get everything right.
Edit: And of course, it's my opinion that all the models are currently suffering from a lack of input data. The "...and what should be done" part of the headline, in my opinion, should be answered with, "everyone download PressureNet and contribute your phone's sensor data to weather models", so that we can better predict difficult quantities like heavy snowfall and severe thunderstorms: https://play.google.com/store/apps/details?id=ca.cumulonimbu... ;)
Do you know of any research about the accuracy of forecasts as a function of sensor data input? I.e. what kind of an improvement will we see if some number of Android users install PressureNet?
The early results are good. There will be an improvement [1]. How much of an improvement? We don't know yet. Cliff thinks it could be a revolution for some types of forecasts, but we don't have the density of sensors yet to know for sure.
For some comparisons, we sent cliff about 20,000 measurements per hour for his experiment in that paper. We're now delivering about 200,000 per hour to researchers, and that's not nearly enough. Our aim is for 2M per hour and I hope to reach that in the next 2-3 months. Around 1-2M per hour is probably sufficient to provide the "revolution" in accuracy that Cliff predicting.
I should also note that 1-2M per hour is small. We should be able to get closer to 1B per hour, but it'll take a while to ramp up to that kind of scale (that'll be like, every smartphone + watch + car that has a barometer).
The improvements will be slow and steady until we get massive scale and are able to run our models in real time. Until then, it's tough to guess how good the improvements will be.
[1] In the linked paper above, I believe the results were a reduction in root mean square error of about 1deg C for a 3-hour temperature forecast in the Pacific Northwest.
Alternative hypothesis that also fits the evidence: You look bad if reality turns out worse than your forecast. You look less bad if reality turns out better than your forecast, thus there is a built in incentive to make forecasts more pessimistic than is justified.
Imagine if 10in were forecast and then NYC actually got 2 feet. Then everyone would be complaining that the city wasn't adequately prepared for the 14 extra inches they received and would blame the forecasters for all the economic consequences of not being prepared.
If I'm understanding the article correctly, this seems correct. Of the many models almost everyone chose to emphasized the models with the more extreme output. Different parties had different motivations: selling news, avoiding political blame...
Interesting things happen when science meets human nature. Science rarely wins especially if there is uncertainty.
> Science rarely wins especially if there is uncertainty.
I agree 100%. One of the most interesting books I've read lately has been The Black Swan by Nassim Taleb, which deals with this at length (ostensibly it is about unpredictability in financial markets, but the implications are far broader than that).
>Imagine if 10in were forecast and then NYC actually got 2 feet.
i'm imagining it, and not really seeing the problem. 10" and 2' are both enough snow that you probably don't want to drive in it unless you have to. It might make a difference for the snow plow operators or the skiers, but for the average person the only thing they need to know is "should i plan to drive to work in the morning" and the weather forecasters correctly predicted "no".
Agreed if you can only report one figure, but the point of the article is that forecasters did only report one figure when they should have made it clear that there were competing models with very different predictions.
After watching the same ridiculous news reporting touting "historic", "storm of the century", "unprecedented", etc., year after year, the more cynical side of me suggests that there's a small supplemental reason besides flawed models: there's probably good money to be made from overhyping storms. At the very least news stations must get great ratings the day or two beforehand. Note that this doesn't preclude accurate estimating...you just have to pull a bunch of cheap psychological tricks to get people to focus on the worst case scenario. The end result of people freaking out is still largely the same.
One reason this comes to mind is that I can't ever remember a storm reaching its worst-case snowfall predictions, and usually they barely make the minimums. A rule of thumb I use is to treat the minimum prediction as the actual maximum and zero as the actual minimum. Anecdotal, but seems to work well.
The "play up the fear" stuff around weather reporting in recent years is why I've given up even attempting to use weather forecasts from anywhere but weather.gov which is sourced directly from The National Weather Service and NOAA. It's just a straight up information dump, no annoying ads or fear mongering for clicks. When I heard they were naming blizzards last year, and this year now, I was only momentarily shocked because it was the next logical step.
This has far more severe consequences then just clickbaiting though. When Hurricane Irene was moving up the coast a few years ago they did the typical "storm of the century, omg a billion people might die!!!" routine and we got light wind and rain. The following year they did it again with Hurricane Sandy (I really hate when people call it Superstorm Sandy ... it was a small hurricane! We were just ill-prepared for it) and a whole lot of people didn't buy into it and paid the price. I'm only 27 but I have trouble remembering a time where every thunderstorm we got wasn't played up to be an extinction level event. At this point I have my small cache of supplies and otherwise completely ignore storm predictions for the most part.
I didn't realize that. My only exposure to weather reports at this point is actually just going to weather.gov and saying "ok, might get snow, I'll go the store tonight instead of tomorrow". And stuff like this is precisely why.
Several years ago in Australia, a blogger called Possum Comitatus, aka Scott Steel, used to mock journalists for breathlessly reporting every twitch of the polls, even though most such movements were within the margin of error.
So successful was this mockery that the way polls are reported actually changed. It become normal to report a change in polls with a remark about whether the change was within the margin. Political race-calling has moved onto less abuses of statistics.
Better reporting can be done, and done with surprising ease, using a simple tool.
Mockery of journalists.
To gain status and prestige over their peers, some will adopt the better method. In a few months they'll all do it.
1. Uncertainty isn't actionable. A statement like, "There's an 85% chance of snow with accumulation of 1–24" with peak probabilities of 28% and 17% at 2" and 18", respectively" is nonsense to most people. And moreover, it doesn't mean anything to me. Are we going to get snow that will affect my morning commute? Are we going to get so much snow that I need to take emergency action? A probability distribution doesn't answer those questions, even if that's the best that exists.
2. Weather is news. The cable news channels were dominated by snow predictions yesterday, and the Weather Channel exists as an "entertainment" venue, not a source of scientific information. If it doesn't fit into a soundbite, the public can't absorb it. And "Weather is hard to predict" doesn't garner viewership.
3. Prepare for the worst. While small changes in the environment can lead to large variability in snow accumulation, the fact of the matter is that there was a not-unlikely chance (according to the models) that NYC, Boston, and other cities could have seen crippling snowfall. They were able to prepare for that eventuality. That it didn't come to pass is almost a non-issue. Can you imagine the outcry—not to mention the impact—if the situation was reversed (i.e., surprise 3' of snow)? Fairfax County Public Schools outside of DC got slammed in the media for not closing a few weeks back.
The real question is: at what point of probability do you prepare for a potential outcome, and are you prepared to back that up (either way)? Do we prepare for record snowfall on 80% likelihood? 50%? 10%? (And as a broader point of discussion, this same calculation applies to other areas: TSA spend vs. terrorism; flood insurance cost vs. coverage; etc.)
There is one practical uncertainty action that can be taken:
I mostly get my forecast from a bookmark on weather.gov, and the "area forecast discussion" is one more click away. The technical discussion isn't any more complicated than sportscasting news once you get used to it, but that stuff doesn't matter, the important part is to read for "FORECAST CONFIDENCE IS HIGH." and "MODELS ARE IN GOOD AGREEMENT WITH" at which point you can pretty much trust the forecast for that time period, and if the textual commentary is all what wikipedia calls weasel word phrases like "SHOWING SOME DIFFERENCES" or "BUT MAY NEED TO BE RAISED IN LATER FORECASTS IF MODELS REMAIN CONSISTENT" means you need to check back in half a day or so, till they firm up a bit.
The only verbal commentary you'll get in mass media reporting is how great lasagna recipes sound and how weather will impact the local ball team, which sucks.
If AFDs could be a little more formalized by date, you could do automated textual analysis pretty trivially and produce an honest forecast. Sometimes weather guys can give you an accurate forecast for next thursday and sometimes they can't, but the monkey in the street DEMANDS some answer for next thursday, even if there really isn't one, even if it makes the weather guy look dumb. An interesting startup-ish idea would be a mostly automated honest weather report app where days when the best scientific minds output a question mark, are displayed as a question mark instead of just makin stuff up. It would be a nice difficult problem, sounds all startup-py to me.
the fact of the matter is that there was a not-unlikely chance (according to the models) that NYC, Boston, and other cities could have seen crippling snowfall. They were able to prepare for that eventuality. That it didn't come to pass is almost a non-issue.
The thing is, for the Boston area at least, the crippling snowfall totals came true. Areas west of Boston were reporting well over two feet of snow earlier in the day.
Yeah, southwest of Boston we are getting wrecked. Still coming down hard, hoping it finishes up soon. I'm glad we had the warnings, and I'll be particularly glad if we lose power, which seems pretty likely at this point, since the warning gave me the chance to stock up, charge battery backups, etc.
The situation with Fairfax was quite different. The forecast there was pretty accurate: a couple inches of snow, falling on warm ground with rapidly cooling temperatures. To someone who understand this, that means the roads are going to be absolutely atrocious. Apparently whoever decides these things at FCPS thinks it means "that's not much snow, so we'll be fine."
Agreed: weather forecasting was not the only thing at play. (I live in Fairfax, and it was also worse than forecasted.) This just illustrates what happens when you get it wrong (whether you are a school administrator, meteorologist, reporter, or policy maker).
People don't like it when 14 inches is forecast and the actual snowfall is 2 inches, but they aren't going to like hearing forecasts of 2-16 inches.
That sort of wide confidence interval is more accurate, but it's also basically useless. How does a 2-16 inch forecast help a parent or school district or mayor plan for anything?
Yes, it's hard to plan for 2-16 inches. But that's a fault with reality, and not a reason to give bad forecasts. It's really hard to plan for hurricanes, so let's just give a forecast of "it's going to miss us"! That's much easier to plan for!
Sometimes life is uncertain. The answer is to account for the uncertainty when planning, not insist that the information indicate certainty when it's not actually there.
A single 9x% confidence interval like 2-16 inches gives the impression of a continuous bell-curve distribution, when you might have a multiple peaks based on a couple of simple factors. I wonder if it would be useful to give out a couple of probability buckets. If people heard "20% chance of 10 or more inches, 60% chance of less than 2 inches", would then know to be ready for a lot of snow, but not be surprised if it doesn't happen, or would they ignore the considerable chance of a heavy snowfall?
I disagree. When forecasters know with certainty that there will be 14 inches, that's what they should say. It's definitely more accurate. However, when they only know that it might go either way, they should say "2-16 inches", which is more useful and accurate in the face of real uncertainty. Even better would be "2-16 inches, probably 14".
A school administrator who sees a forecast of "14 inches" may simply cancel class the evening before, giving people time to prepare with certainty. That's nice, but not at the cost of a lost school day and calendar disruptions.
A forecast of "2-16 inches, probably 14" might cause that same administrator to wait and see. TV and radio would broadcast the news about possible school closings the evening before, so parents could prepare. They might have a bit more difficulty planning, but that's worth it if the school day can be saved.
Similar to what this article suggests in its postscript, you wouldn't have to merely say 2-16 inches. Instead, you could say, "there's a 50% chance it'll be between 4-6 inches, and a 15% chance it'll be between 14-16."
We should always strive to improve our predictions, but I see a fundamental dichotomy: do you prefer false-positives, or false-negatives? We don't get to pick no wrong results.
When it comes to predictions like snow storms, hurricanes and tornadoes, you obviously want to favor false-positives. It's better to predict something bad will happen, pay the preparation costs and have nothing happen, than to have something bad happen when you're unprepared.
So, yes, we should try to improve our predictions. And perhaps we should be better at communicating our uncertainty. But I don't blame the meteorologists, or the people in government who shut things down and told everyone to prepare.
Note that this article is written by the Capital Weather Gang, a group of DC-area meteorologists who consistently put out accurate and nuanced short-range forecasts of upcoming weather in the area. They give out great info and they aren't afraid to state their uncertainties up front, nor are they afraid to admit it when they're wrong (which is not too frequent).
They give an implicit answer to the question of "how could everybody do better?" which is "be more like us." And I think that would be great.
I was curious about what kind of bump weather sites get in ad revenue as a result of storm hype.
That's an interesting angle when wondering where the hype is driven from ... if you find the mental image of Murdoch twirling his mustache (this is my imagination, so why not add a mustache?) while making quiet phone calls from a back room a correct version of reality.
How is it possible that millions of New Yorkers accurately discounted the probability of it being a 500-year storm and yet the mayor didn't? The responsibility must lie with either the Mayor or the meteorologists. And there is something to own up to -- shutting down the city is not without real cost to real people. The mayor and governor will stay it was for our own good and better safe than sorry. But put a few million people under house arrest (no using roads -- under threat of arrest, no public transport) for basically no reason.
But a few things to consider:
1) The meteorologists with the most dire predictions are the one who are going to be picked up by the media -- the media needs to sell ads.
2) The bulk of the population accurately discounted the most dire predictions, because of well-calibrated but subconscious Bayesian calculation.
3) Bureaucrats, and to some extent politicians of both parties, have a well-intentioned, but costly and wrong, need to subvert the judgement of millions and replace it with that of a handful of experts. When the judgement proves wrong, they rest on their good intentions without assuming responsibility.
4) In subverting these judgements bureaucrats and politicians can and do rely on the states monopoly on force.
It seems like the costs are asymmetric on misses. If a bunch of people die, the mayor gets voted out of office. If they prematurely close the city for a day, the mayor just apologizes afterwards.
So the line on the map of where the heaviest snow would be was a bit off: the only reason this is "news" at all is because that line happened to affect New York City, and there are a lot of journalists who live there whose view of the world matches that New Yorker cover[1]. If the line had been anywhere else we wouldn't have heard about it at all.
[1] https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Ave...