He seems to have done a pretty good job of negotiating his contracts with respect to owning a lot of his model IP moving forward. Still needs to find a platform/pub moving forward of course but far from starting from scratch.
> Much of FiveThirtyEight’s vital intellectual property — such as the election forecast models — is merely licensed to Disney. The license term for these models expires with my contract this summer. I still own these models, and can license or sell them elsewhere.
This is one hell of an agreement to have with your employer. That was my next question given that most of FiveThirtyEight was Nate's own work, and he had very good insight to have an escape hatch.
I really hope to have the power to do this one day.
What interest would the Mouse have in numerical models? They want entertainment IP, not weird programs that they have no experience marketing or hiring people to run. They’re not just going to transfer it over to Pixar so the guys behind Cars 2 can predict the next election.
Disney wants the talent. It’s just not as valuable to them as keeping Nate happy
Disney is a multi industry conglomerate interested in understanding/manipulating public policy and political trends through numerical models in the same way you would be interested in your local politics and it's trends.
They're also not just after "entertainment IP". In fact I'd say they are one of the premier proprietary data analytics orgs with respect to park management, fast passing, queuing. Same with Market Research on film/animation, they've quite literally been doing it since it's been a thing.
I wouldn't presume that most of it is Nate's own work. (I suspect that for most sports he is delegating but "owns" the output, he does seem to directly build most of the political stuff.)
Isn't this what the Skype guys pulled off with Microsoft too?
Nate got his start in this world doing baseball forecasting (https://en.wikipedia.org/wiki/PECOTA), well before he delved into politics. Jay Boice definitely doing much of the heavy lifting these days, and deserves more personal recognition than he currently gets. But I wouldn't discount the amount of work Nate personally put in on the sports front.
I don't entirely disagree. We know from his public statements that he personally builds the political models. I think he is personally invested in the baseball modelling but we know it is a team effort and 538 is doing a lot. Nate is a media figure, he has to delegate.
Wow... if that's true, it's some real amateur hour idiocy. As much as I like the brand, I'd trivially stick with Nate, especially if he somehow got the old band back together: Harry, Claire, etc.
I'm not sure how much I'd read into that. Maybe some Disney execs have seller's remorse (or never wanted to sell in the first place). But, at the end of the day, a lot of the "IP" is pretty much public in considerable detail. (That said, having some explicit ownership is a good way to avoid unnecessary haggling with lawyers.) And if Disney had actually wanted the full analytic and writing ability of FiveThirtyEight surely even an entertainment exec would realize you're giving that up if you let those people go.
> There are also some complicated cases — for example, for some of the sports models, I own them, but Disney gets to keep a copy when I leave.
As others have mentioned, he's very wise to have negotiated contracts so he keeps some IP, but those sports models might be what generate the most revenue.
Are there precedents for this in the tech world, where somebody can legally bring IP with them to a new role but the old employer/competitor keeps an exact copy? Seems like an unusual arrangement that'd be hard to sell the upside to somebody competing with Disney/ESPN.
> Are there precedents for this in the tech world, where somebody can legally bring IP with them to a new role but the old employer/competitor keeps an exact copy?
This is actually the standard arrangement, depending on how you define "tech world". In California for example a Proprietary Information And Inventions Agreement will include a provision along the lines of:
> If, in the course of my employment with the Company, I incorporate a Prior Invention into a Company product, process or machine, the Company is hereby granted and shall have a nonexclusive, royalty-free, irrevocable, perpetual, worldwide license (with rights to sublicense through multiple tiers of sublicensees) to make, have made, modify, use and sell such Prior Invention. Notwithstanding the foregoing, I agree that I will not incorporate, or permit to be incorporated, Prior Inventions in any Company Inventions without the Company’s prior written consent.
Meaning anything you created before remains yours, even if, out of the kindness of your heart, you allow your employer to benefit from it.
What's unusual in Silver's case is maintaining that arrangement for a) the founder of a company, and b) through an acquisition.
How much value does the model actually have, in and of itself? It’s just a bunch of correlations, right? The value of the model is in the reporting that he does using it (articles, podcasts).
The competition would get value from having him host a show about it on their platform. Disney… I wonder if they even care about the model, or if they just took a license to it/current copy thing because, I dunno, maybe he offered it for really cheap.
I mean can you imagine, 538 but without the main character? It would be incredibly lame.
> It’s not just that I think these models have a lot of journalistic value
...do they, though?
I'm actually not so convinced in the journalistic value of FiveThirtyEight. It's something I've thought about before.
FiveThirtyEight's political forecasts absolutely provide a desirable service! Who doesn't want to know who will win an upcoming election?
But is it good for people to know who is more likely to win an election? Are these predictions beneficial to society or democracy? I sometimes think they may actually be harmful, decreasing voter turnout by convincing people (correctly or incorrectly) that the result is preordained.
That's not to say Nate Silver is doing anything wrong by making FiveThirtyEight available, but he tends to talk about it as a sort of public good. I'm not sure whether that's accurate.
If anything, I suspect this detailed polling of making elections closer. Millions of voters aren’t reading 538. It’s a distinctly wonky thing to do. But people really into politics, the kind of people who volunteer for campaigns and make large contributions do pay attention and I think it helps rally their support behind realistic candidates for their party so that the candidates put forward for the general elections are as closely matched as possible. Well funded campaigns probably already had their own private polling and statisticians, but 538 helped make sure all highly invested individuals had access to decent data and could deploy their resources more effectively. If you keep that data out of grassroots hands, then you put the parities more firmly under the control of a very small party elite with all the experts.
True. But millions of them will hear a radio announcer, or TV person, or tiktok video or whatever else say something like "In the latest polling, <X> has pulled ahead of <Y> ..." where "the latest polls" will mean (among other things) 538.
I do believe that this has some effect (if small) on patterns of voting (particular by encouraging or discouraging turnout).
Generally speaking, you don’t hear this because 538 is not a poll, it’s a political analysis brand and competitor (ABC news). Most news orgs or journalists quote the polls themselves.
And that will happen no matter what. Polls and prognostications will always exist and probably have always existed back to the first tribal chieftain election. It’s just that instead of getting totally inaccurate predictions that only serve a narrative, for a little while a few of us knew how to get a more unbiased prediction. Now even fewer people will know anything aside from their favorite outlet’s media bubble.
I don't think dragonwriter was weighing in on your broader point, but being a little pedantic (and a lot correct) about what work is done by what people in our electoral coverage.
It's generally not understood to have positive connotations. In part because there is indeed some evidence to suggest that your concerns about it harming the democratic process are valid. For example: https://www.journals.uchicago.edu/doi/10.1086/708682
In fact, horse race journalism has been observed to have such negative and self-fulling effects that some countries ban polling and/or electoral journalism for hours, days, or weeks before an election. Even in the US, it is illegal to do any "electioneering" within 100ft of a polling place.
On top of that, while 538 initially provided good and solid insights, it has recently been successfully gamed by right-wing players by standing up bogus polling organizations and getting their bogus results into the aggregator's (like 538) results. The goal of course was to make their candidates seem more popular, and the "reliability" scores did not work, as the aggregators results were significantly off in that direction.
> On top of that, while 538 initially provided good and solid insights, it has recently been successfully gamed by right-wing players by standing up bogus polling organizations and getting their bogus results into the aggregator's (like 538) results. The goal of course was to make their candidates seem more popular, and the "reliability" scores did not work, as the aggregators results were significantly off in that direction.
[Citation needed]
Lots of people raised that concern at the time, predicting the effort would bias aggregators, but for 538 at least, I’ve seen no analysis indicating a significant swing to a right bias in 2020, and overall the 2020 results were quite accurate.
I was watching it in real time. The effect was mostly in 2022 (they hadn't really gotten it going in 2020)
Here's the 538 headline before election day about how the Republicans were in very good shape, even potentially a blowout win [0]
In fact, it was the exact opposite.
Yes, there was a lot of statistical handwaving about how they were really right, but, not really. BTW, this was just the top result from a DDG search, and is only one of the things I remember.
FiveThirtyEight is beneficial, and it’s fairly simple as to why.
1. People want to know who is winning. It doesn’t matter whether you like it or not. People want to know, and thus there will always be a market for it.
2. Polling is imperfect and it’s a lot of work to understand how to interpret the various polls out or to forecast in lieu of polls.
Political journalism before FiveThirtyEight was filled with even more crooks and charlatans coming up with nonsensical ways to forecast elections or interpret polls. There was a lot more noise in the system. FiveThirtyEight shut that stuff down to an incredible degree.
3. FiveThirtyEight has introduced more folks to probability theory than I think any other journalistic collective. What has always made them different is that they provide probabilities, not certainties.
Societal value != journalistic value. And in fact, even if there's a net negative societal value that doesn't mean it shouldn't be around.
Nate silver didn't exactly invent political polling either. That's doing a disservice to the enormous time and money invested in polling over several decades of elections (and the learnings when things didn't go correctly). Polling will be around regardless.
That's fair. I was responding to ways I've heard Nate Silver talk about the model and polling in general in e.g. podcasts, which is why I've had the question floating around in my mind. But you're right, that's not quite what he wrote here.
> And in fact, even if there's a net negative societal value that doesn't mean it shouldn't be around.
To be clear: I don't mean to imply that FiveThirtyEight shouldn't exist. There's lots of things which serve a human desire but aren't necessarily a social good. Slot machines for example.
The big value of polls is a backup check for discrepancies. A major difference between final polls and polls leading up to the election could be innocent (like unexpected demographics), but it could hint at voter suppression or even completely innocent issues with the voting systems. Paired with exit polling, there's enough information to find answers.
As opposed to horserace election coverage, which is the single biggest focus of all of American journalism for the year or so leading up to an election... which is of greater journalistic value?
Simply looking at 538 during election cycles makes it hard to argue they aren’t a horse race tracker (they literally started as a sanitized gambling odds site and Nate’s new book is about gambling).
They’re just the guy who went to the stables before to see who was eating the most hay.
It's really not, which is why quantitative models have real value. You don't have to argue that he's biased, you just have to point to the evidence. But your whole comment is indistinguishable from someone with sour grapes who is simply casting aspersions on his track record without being willing to analyze it properly. At the very least, you ought to specifically identify what kind of biases his predictions have shown; he leaves himself wide open to being proven wrong, so it's only fair for you to do the same.
Nate put 2021/22 warriors at 0.5% to win, Vegas was +900.
Nate put 2020/21 bucks at 10% to win, Vegas was +550.
If you can’t even come close to beating the line, you’re not a good model. He’s just transparent about his variables.
Edit: To be clear I don’t gamble, I just follow sports and like data analysis, so this isn’t a bank account thing.
Which non-538 models consistently outperform 538's models?
EDIT: (cause I can't reply to the below comment) my question was what model beats 538, not which betting market beats 538. yeah, duh, the Vegas line beats 538, because it beats literally all publicly available models. It'd take a really disastrous gambling market failure for anything else to happen.
Nate put 2021/22 warriors at 0.5% to win, Vegas was +900.
Nate put 2020/21 bucks at 10% to win, Vegas was +550.
If you can’t even come close to beating the line, you’re not a good model. He’s just transparent about his variables.
> EDIT: (cause I can't reply to the below comment) my question was what model beats 538, not which betting market beats 538. yeah, duh, the Vegas line beats 538, because it beats literally all publicly available models. It'd take a really disastrous gambling
Lines aren’t built that way. They’re set by books to offset their risk. They’re based on a bunch of (usually drunk) wetware computers with huge biases (don’t believe me, look at any Dodgers line when they have an obviously bad pitcher matchup).
If your model is worse than that, then what’s the point of a model?
To the point of bookie models, those are based on how they think people will bet, not on who they think they will win. The point of books is to be outcome neutral.
The models are one thing. It's the data that really matter imo. No other organization has created as thorough of a collection of polling firms, their polls, as well as their faults and past records as FTE.
Even if you're building your own model you'll likely be turning to FTE for the latest data.
In addition they're data driven journalism is often paired with amazing data sets that they open-source for free.
> who doesn't want to know who will an upcoming election?
In particular, politicians want to know. This to me is the interesting upshot. Polls are non binding, so the public could use them to signal to candidates how to tweak their platform.
I’m always curious if there’s any real good data saying turnout patterns change. The thing is, voting because you think your one vote might change the result, is already completely irrational. I always think of voting of being like when you go to a concert and buy the band’s CD even though you already have Spotify: it’s more about find a way to personally participate in an important life moment than practicality.
I think the major parties and other interested organizations will always have forecasts like these available internally, and will leak them (or lies about them) when it's to their advantage. Having a high-quality forecast directly available to the public can't hurt more than that.
> I sometimes think they may be harmful, decreasing voter turnout by convincing people (correctly or incorrectly) that the result is preordained.
This is an old tactic that is used to the hilt by both sides whenever it serves them, and media outlets already make forecasts, often without 538's scrupulous emphasis on how often they're wrong. I don't think 538 makes this worse.
> FiveThirtyEight's political forecasts absolutely provide a desirable service—who doesn't want to know who will an upcoming election?
> But is it good for people to know who is more likely to win an election? Are these predictions beneficial to society or democracy?
Two points:
1. The model DO NOT tell you who will win the upcoming election.
2. It is absolutely beneficial to society and democracy, having a poll-free & rigorous-analysis-free democratic society means an environment where even the most apolitical citizen can be bamboozled into thinking that an election was "stolen" simply based on their close associates and news sources. Polling and aggregation, when executed with a modicum of accountability, prevents a devolution into a Soviet-style culture of paranoia particularly among the elites. It has tremendous value, that value will be backfilled by rumour and innuendo and that is very bad for society.
> But is it good for people to know who is more likely to win an election? Are these predictions beneficial to society or democracy? I sometimes think they may actually be harmful, decreasing voter turnout by convincing people (correctly or incorrectly) that the result is preordained.
For plurality voting (a.k.a. first-past-the-post[0]) with more than two candidates, voters have to cast their ballots strategically[1]. This is especially common in primary elections and non-partisan positions.
Unless all elections switch to a method that is not subject to the spoiler effect (like approval voting or instant runoff voting), voters need more information about which candidates are most likely to win, not less information. Sometimes candidate competitivness can be inferred from things like endorsements or fundraising numbers, but polls -- and aggregated models based on polls -- are an important tool.
It has been fairly useful to me as an observer and voter in understanding how the elections go down. Understanding what the chances are of each candidate and how certain voting groups can influence the election can (sometimes) reduce being blindsided by an election. It may reduce my impartiality as a voter, but it definitely helps me process politics as a citizen.
Even if he result appears dead-even there is a cost. Framing the election in terms of who wins instead of which candidate is best increases the tendency of voters to vote for candidates who "have a chance," even though there is no rational reason to vote for candidate that is more likely to win.
I would think forcing people to better appreciate how probabilities actually work is better than avoiding the topic entirely. Learning that 70% is not certain is important in lots of contexts. 30% probability events happen all the time…
Data illiteracy, not the existence of a forecast, is the core problem.
FiveThirtyEight has had astonishing accuracy since 2008. They don’t pick winners, they provide data analytics of polls and other factors. People may not appreciate probability theory but this is far more reality based than what passed for political analysis prior to 2008.
After watching this and the DPReview saga, there is only one takeaway:
Don't sell your company unless you are prepared to walk away and do something else - because at some point you're going to be forced to do so whether you want to or not.
Or do exactly what Nate Silver did – negotiate the contract in such a way that you still have control over your IP if they decide to terminate you or sunset your product. Of course chances are that you don't have the kind of leverage that he did, but still...worth a shot.
> "In July 2013, ESPN (owned by Disney) acquired FiveThirtyEight, hiring Silver as editor-in-chief and a contributor for ESPN.com; the new publication launched on March 17, 2014."
> "BB-8 was first seen in the 88-second The Force Awakens teaser trailer released by Lucasfilm (owned by Disney) on November 28, 2014"
> Much of FiveThirtyEight’s vital intellectual property — such as the election forecast models — is merely licensed to Disney. The license term for these models expires with my contract this summer. I still own these models, and can license or sell them elsewhere.
Sounds like he had some good leverage going into this deal. Why would Disney have agreed to a temporary license, as opposed to a purchase of this critical IP? Regardless, good for him.
> Why would Disney have agreed to a temporary license, as opposed to a purchase of this critical IP?
It was probably a risk hedge from them.. he would have wanted a lot more for the full enchilada and they didn't know how it would perform in the future. And now his work is not hard to reproduce so it wasn't a bad call. Every political data team I have heard reporting on essentially concedes that whatever they do ends up more-or-less matching 538.. so it is hard to extract surplus value but also easy to hit that mark roughly.
I follow the NHL and there are a lot of models, they disagree in certain details but they largely agree and the "winner" at the end of any given season isn't winning by much.. even private data isn't a big boost.
And who knows how much of this Disney really cares about going forward outside of the Nate Silver Brand--and had they had wanted that they'd have kept Nate Silver.
Calculating the difference between the 538 brand and the Nate Silver brand at the time of acquisition was probably a bit of a toss-up too. Nate can move on to a substack that generates $2MM a year with 4 staff and be happy while Disney runs 538 at a reasonable margin and be happy and I guess everyone is a winner?
In the meteoric rise of 538, Nate could have easily walked into NYT or any other tech-forward media outlet and negotiated to retain the models. Disney would have lost out then.
Disney and WB Discovery seem to have totally lost any vision for their future.
538 has a very clear value proposition outside of exposing nerdy polling and forecasting breakdowns for a small group of wonks. It's the place a lot of other folks source for rigorous tracking+forecasting of elections, sports, etc. It's good.
Disney, however, seems to be putting out a lot of crappy content on their Disney+ platform that rates poorly and is expensive to produce. The Mandalorian + Andor are great but the other recent Star Wars IP is horrible and cost them a fortune.
As a DIS shareholder, I'm livid. Losing the rights to use the best election forecasting models in the business a year before a Presidential election is monumental fuckup.
This is one screwup that can't be placed at the feet of their previous incompetent CEO.
In the grand scheme of Disney stuff, it probably didn’t make much money relatively speaking, but it seems weird to get rid of it right before the season where it’ll make any money.
It’s not that they were wrong, they weren’t useful for what people want. What people expect and what is possible is out of sync and I think people misunderstood FiveThirtyEight for many years and thought it was an oracle. It’s not.
A melancholic development. But in a way, kind of relieving.
I think Nate Silver and his team did an amazing job making statistical analysis interesting and transparent and trustworthy.
On the other hand, 538 was a weirdly sterile website to actually read. On the rest of the internet you have raging and bombastic news designed to elicit feelings and (most importantly) clicks. 538 tried their best to fit in this world, but you were still just reading data analysis about other news or topics.
> Much of FiveThirtyEight’s vital intellectual property — such as the election forecast models — is merely licensed to Disney.
This seems odd that Disney would buy FiveThirtyEight but not its IP. How common is this?
If I were buying a data journalism company known for its models, that seems like something important to purchase. This seems like a curious contract negotiation and seems like a mistake on Disney’s part. Or a future lawsuit if there’s a difference of opinions.
Disney is a media company that was dabbling in data science because of its name recognition. It almost certainly didn't have the strategic vision to care much about something as nerdy as the intellectual property being used to generate that name recognition.
FiveThirtyEight made a big splash in its 2008 election debut. According to Wikipedia, there were numerous media outlets competing for its content in 2010. That likely gave Silver a lot of leverage. The same is probably true in 2014, when FiveThirtyEight moved from the New York Times to ESPN/Disney.
Interestingly, the deal with the NYT was that the entire blog was licensed. For the deal with Disney, it seems like Silver gave up a lot more, and only kept control of the models.
I mean, 538 was under the NYTimes.com domain before that, it's not like Nate had to sell, and when he agreed to move to ESPN/Disney, he surely had more of the bargaining power.
I don’t think anyone can answer this to your satisfaction without knowing your personal definition of journalism. Because I believe the question you’re really asking is, “Was 538 what I consider journalism?”
When I typed “define journalism” into Google, this came back:
> the activity or profession of writing for newspapers, magazines, or news websites or preparing news to be broadcast.
So yes, 538 was journalism by at least one definition of journalism.
Here’s one more definition:
> Journalism is the activity of gathering, assessing, creating, and presenting news and information. It is also the product of these activities
I used to scrape craigslist for local offers. Many times I needed to be the first to call, so I wanted my script to get the numbers.
In about five minutes I came up with an algorithm that defeated most of the obfuscation tricks of the time:
Strip whitespace, convert "five" to 5, remove special characters, and look for ten consecutive digits in the body text. Maybe a couple of small tweaks after that like removing text between digits.
Most people, when they invent their little unique scheme, invent one that is already defeated by the algorithm above. At least for phone numbers.
Anyway, all that to say you're right, it never worked.
Spammers aren't that smart, they will absolutely go where the bar is at its absolutely lowest, if not missing entirely. Just changing your SSH server to operate a port that is not 22 is going to stop most if not all random login attempts.
I wouldn’t characterize that example as “not smart” to be honest. I’m not particularly smart, but if someone has gone through the trouble of changing their secure socket port from 22, I would realize there would be more work and time involved in finding the actual port, and then it would be a safe assumption that, if found, it would have significantly more security hardening than the average port 22 because clearly the admin knows enough to change ports. So I might even characterize it as smart to not go after ssh ports that are not 22.
The reason is different. When you need to scan 5 million IPs on 22 port is one thing, but 5 million IPs times 100 or 1000, then you simply run out of resources and since 99% of people use 22, why bother?
particularly odd when natesilver.net and natesilver.com are both domains that exist and presumably owned by him (for well known public figure trademark reasons), he can't just point the MX for them at gsuite and then set up an inbox?
That opens a surprisingly large landscape of possible attacks and failure modes. The jsit obvious subset being things that happen when your paid gsuite gets disabled (not paying, abuse report, etc), or the domain lapses or is taken over.
A basic free gmail account is very stable compared to almost any other option. L
somebody of the public profile of Nate Silver presumably can hire IT/network people who know how to securely run authoritative DNS (or hire it out to a highly trusted third party as a service), so they can point the MX anywhere they want, and either do email in-house or hire it out to an email service specialist or big cloud service provider.
The risk of having the gsuite paid services shut off are about the same as any other high dollar value paying customer using gsuite or office365 as an email service.
The question was why not, and the answer was because it's more risky. Why would Nate Silver want to do all the things you correctly listed as options when he can simply use the safer, more simple, free gmail option?
I think Nate Silver must be some kind of punching bag in right wing media circles, because some variation of this comment is posted basically every time 538 comes up on HackerNews.
This is more of an average person/mainstream media reading complaint. But they weren't paying attention in 2016 or else they would notice /he was the only person saying Hillary might not win/.
He is the face of election prediction in America. All the other guys at NYT, WaPo, etc aren't as famous. Of course he's going to be the face you punch for them not predicting the election remotely right. (And don't give me this shit about 75% or whatever odds he gave being the rightest or whatever weasel nonsense, it was still plenty wrong and he still want to lmao "spike the football" really? Get out of here.).
If there is a 2/3 chance of a thing, and you say there's a 2/3 chance of the thing, you're doing your job right even though you "got it wrong" in hindsight 1/3 of the time. If you want to assess how someone is doing at prediction, you need to look at a lot of predictions and if they're giving probability estimates you want to apply a proper scoring rule. If you're going to make grand pronouncements based on a single close race being labeled the wrong side of close, get out of here.
> We also received a lot of criticism from Democratic partisans in the closing weeks of the campaign — more than we did from Trump supporters — because they thought we didn’t have Clinton as a heavy enough favorite. That’s unusual. We’ve forecasted enough races over the years to have taken criticism from almost every side. But in the past, it’s always been the trailing candidate’s supporters who gave us more grief.
Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times?
The accuracy of the polls is another matter entirely. You could estimate the chances that your polling was wrong, given that the selection of the sample is a random variable, people lie, and intentions change, but that's not at all what the general public thinks the number means.
> Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times?
You can calibrate it by looking at their other election predictions and seeing how generally on target it is.
no, you can only "calibrate" his prediction abilities in general:
"given 100 predictions by Nate Silver, how many will be correct?"
that's what you just said.
In other words, if we forgot what actually happened, picked one past prediction out of a hat, asked "so was he right?" and then checked the results, he would be correct 95% of the time. That's what you're saying.
Correctness is actually a pretty bad evaluation because it reduces the outcome to a binary. There are better measures like a brier score that IIRC, they have pretty rigorously evaluated on their own predictions.
It's telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. Why?
There are many thousands of MLB games. There are a few thousand US House elections. There are only a handful of Presidential elections. It is not possible to measure their accuracy on those with any statistical rigor.
> It’s telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article.
All the types of predictions 538 does are covered in the Brier skills chart on the overall summary; there’s detailed analysis covering each type of prediction (with further available breakdowns for subtypes available once you’ve gone into the type) available from the dropdown at the top of the page.
OK, I missed the dropdown at the top. There is not "detailed analysis" for Presidential elections, although I guess you mean that each state is a data point.
All that graph shows is that their "probability" is roughly in the ballpark with huge errors in the middle of the range, which is probably the swing states we care about most.
I'll suggest one concrete case where their probability of a candidate winning a 2-person race has some exact meaning:
*You'll pick a number randomly from 1 to 100. If it's less than or equal to candidate A's "probability" of winning, then you bet $10,000 on A. Otherwise you bet $10,000 on B* (we need to put real skin in the game)
Are you willing to abide by that rule? If not, you don't really believe the probability.
> There is not “detailed analysis” for Presidential elections
There is, in fact, detailed analysis for Presidential elections.
> although I guess you mean that each state is a data point.
No, each individual prediction of each of the 56 (enumerated upthread) distinct elections for one or more Presidential electors is a data point.
> Are you willing to abide by that rule? If not, you don’t really believe the probability.
Well, no, not being willing to abide by that rule means either not being rich enough that winning and losing $10,000 are roughly symmetric in utility or not being an SBF-style nutball and demanding a non-zero risk premium. As it turns out, I am in both categories.
what does this mean, "each individual prediction of each of the 56 (enumerated upthread) distinct elections"?
how many predictions are there for each election? does 538 issue more than one per election? Or how many data points are there?
as for the last paragraph: I guess you don't really believe in their probabilities, then, which was my whole point. I have no idea what you're talking about, re "non-zero risk premium."
> how many predictions are there for each election?
One for each batch (which may be as small as one) of data (polls specifically in the polls-only model) added to the data; each takes into account current and past polls, with time.-based decay in weighting, distance from the election, and other factors. This includes polls for other contests in the same cycle, because the models accounts for correlation between them.
> does 538 issue more than one per election?
Sometimes multiple in a day for an election (especially the Presidential general election.)
> Or how many data points are there?
Well the scale for the bubbles on the Presidential election night calibration plot runs from 30-300 per bucket, and there’s 21 buckets.
> I have no idea what you're talking about, re "non-zero risk premium."
If you don't understand risk premiums and the asymmetric utility of nontrivial quantities of money, you have really no business talking about when a bet is reasonable.
OK, so maybe we're agreed: you can bet on his abilities in one election. Let's say he's been right 95% of the time (I don't know if that's true) and we believe that's likely to continue, knowing nothing except "this is an election and Silver's predicted the result."
Then if he says "Hillary is likely to win" we can have 95% confidence he's right.
If he says "Hillary has an 80% chance of winning" we ignore the 80, and just observe that it's more than 50.
It's a bit more flexible than that, though - rather than ignore 80% vs 70% and make it all up or down, you can let him predict a few different events in a row, add up the errors, and see how off they are.
Or if you do want to review the past, you can look at the error for a category of elections or that entire year rather than his whole prediction career.
> One could note the number of times that a 25% probability was quoted, over a long period, and compare this with the actual proportion of times that rain fell.
it still depends on many samples, or "over a long period" in your doc.
You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
> You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
That depends on what question you're asking. "How well calibrated are the electoral predictions that FiveThirtyEight makes?" is a sensible question with a lot of data points, seems to speak directly to the crowing about the one call being bad, and seems well suited to the application of a scoring rule for comparison between people making predictions about the same things.
With a straight face you’re going to say that in 2016 you saw a bunch of polls coming out showing trump winning?
Absolutely not. Nobody, and I mean nobody, gave him a snowflakes chance. Everyone including Silver mentioned “the greatest election landslide in history”, or some variant.
If there were polls showing trump winning they were hidden from ans downplayed in the msm. And to say anything else is a very poor lie.
The last poll there from Nov 8 showed Clinton winning the popular vote by...wait for it...3.9 percent. She won the popular vote by...2.9 percent. National polls weren't far off at all.
The electoral college was decided by less than 50,000 votes overall. No one predicted the margins in those key states would be so small. It was a razor-thin win.
Statistics always have a margin of error and give probabilities. Polls didn't have to predict Trump winning. Instead, they give a probability of winning. FiveThirtyEight predicted a 28% chance of winning[2]. That's based on Monte Carlo simulations (that's where they run, e.g. 100 random elections and count how many times each outcome occurred). It actually doesn't matter if that estimate was low; it was high enough (1 in 4 chance) that the outcome wasn't insane. Even three days after the election, they already had an analysis of what they went wrong with polling[1]. I mean, you could read that or continue to trot out tired polarized arguments.
I love how the popular vote is only ever mentioned when it’s to defend the idea that all these analyses “were actually right”
You’re not getting it. The analysis, the “28%” - those were wring. those were poor analyses
Yea they came back after and made up some bs about “oh we know what we did wrong now!”
When I was in school all the people who were cheating their way through math did the same thing. They couldn’t solve the problem, and the second you gave them the answer they magically “got it”
Fortune telling is a business of luck. Unless you can convince your followers that you’re right even when you’re clearly wrong
Forecasting a 28% chance of something happening is likely to end with it happening about 1 out of 4 times. At this point you are just being deliberately obtuse.
It's more of a business of being connected to some forms of future reality. I can understand if people think nobody can do better than pure chance, but even so, I'm pretty sure the folks who believe statistics could give them an edge are quite misguided.
This is a very strange take since it was Nate Silver who gave trump an order of magnitude higher chance of winning than everyone else. Around 30% chance, same for a batter getting a hit in baseball, which happens all the time.
I think a lot of people looked at his last prediction that Tuesday morning, observed that 70 > 30, and assumed that meant it was a lock. And then spent the last six years blaming Silver for their own innumeracy.
I find it fascinating that so many people here jump to the assumption that the average person doesn’t get that giving a 28% chance of victory doesn’t mean they can’t win
Clearly. Its not a difficult concept.
But those same people are wholly unable to see that all the other verbiage and propaganda that was behind/infused with Silver was, in no short terms, a demoralization campaign against the trump voters
And since they can’t see that all they can do is project their ego to say they aren’t wrong, its that the people who don’t agree with them don’t comprehend very basic statistics
I don’t remember NYT. But I recall the other major poll site Real Clear Politics had an election map with a percentage chance for each state and they had Hillary at 99%.
Their mistake was that they treated each state as an independent event. Treating each state as a separate biased coin toss leads you to put Hillary at 99%.
Nate knew that these are not independent events and if republicans are being under polled in one state, it is more likely they’re being under polled in all of the states.
Well NYT themselves announced an editorial board overhaul a year or two ago because they said they had become too “left”, perhaps like Fox and WSJ were so far right? But now I’m not sure, since i subscribe and it seems balanced.
Fivethirtyeight had Trump at a 28.6% chance to win on the night[1], the web page is still up. I recall that being higher than any other polls I saw at the time. A 1 in 3.5 chance isn't that unlikely, it implies they're very close.
Percentage probabilities are meaningless for an event like this.
Can you run the 2016 election 1,000 times and count the times Hillary wins? Of course you can't.
Unlike estimating "what's the chance it rains tomorrow?" you can't even find 1,000 elections in the past with roughly similar conditions.
Unlike a baseball season, you don't have enough samples for the true probabilities to come out.
So, yes, despite all your downvotes, you're right: he pandered to the people who want a percentage number, despite the fact that it's meaningless and they will misunderstand it.
This is a lot like saying, "fractional exponents are meaningless, because you can't multiply something by itself half of a time." This intuitive model has broken down, but the problem is that the model wasn't sufficient to capture the complexity of the problem; it was a pedagogic stepping stone.
Similarly, probability is not the result of conducting repeated experiments (though the results of repeated experiments can be used to estimate probability). It's the measure of an event relative to the measure of the set of all events, eg, it's how much of the total volume in the space is taken up by a given event.
If you have a question I'm happy to clarify. If a restatement is helpful, the problem is that this definition of probability is limited and fails to capture the complexity of the problem domain, not that the problem domain is meaningless.
You're misunderstanding the terminology, "events" refers to sets of potential outcomes (eg, when rolling a die, the outcomes are 1-6 and the event "even" is the set of 2,4, and 6).
(I don't know 538's actual methodology, consider the below an illustration rather than a literal description of 538's model.)
The outcomes here would be each way the electoral vote could possibly be allocated, based on the number of states and the idiosyncracies of each state constitution. The events would be the division of these outcomes by who wins the election in that instance. The model then seeks to attach a weight to each outcome based on how likely it is.
You can think of these weights as the volume of each outcome. We use some kind of model to assign them. The most straightforward model would be to repeat an experiment many times and then assume the rate at which the outcome appears in your sample is approximately the same as it is globally. Other times that isn't possible and you need to use more sophisticated modeling techniques. I'm not going to offer a defense of statistics and machine learning writ large, but I think looking around it should be pretty clear these things can work well even if there's a lot of art to getting it right.
You can imagine comparing the probabilities of each event by pouring their constituent outcomes into separate graduated cylinders so that you can compare their cumulative volume. If you divide their volume by the total volume of all outcomes, that's the probability of that event. Or rather, that's the estimate of the probability, given our model and the data we have.
This is more or less my drunk history version of measure theory and probability theory. I imagine I've gotten some details wrong, so I'd encourage anyone interested to check the subjects out for themselves. Here's some YouTube videos:
> Percentage probabilities are meaningless for an event like this.
No, they aren't.
(They can’t be validated for a single event in isolation, but 538 predicts lots of election events, so their models can, generally, be assessed, and are historically quite accurate.)
Yes, I have. I have even explained thr contradiction.
> You just said you like 538.
Not in this subthread.
> Their "lots of election events" are not different samples of a single variable -- they're different variables with different methodologies.
No, its lots of events with the same methodology (within an election cycle, within any class of election events.)
Now, in principal the Presidential general election prediction is a unique (per cycle) event, but the rules for assigning electoral votes based on lower-level electoral results are known in advance, and the national Presidential election is (equivalent to) a mechanical composite of the state level predictions via a model accounting for their degree of interdependence; both the state level prediction and the degree of independence of their variation are testable, and the model does not purport to predict variations due to faithless electors or interventions to assign electors other than by the state election rules in place prior to the state elections.
EDIT: Actually, there’s lots of predictions to evaluate even for the overall Presidential prediction, because there is a new predictiom every time they update based on new polling data, and you can evaluate the accuracy of the model using all of those predictions, not just the final one.
> > You haven't contradicted anything.
Yes, I have. I have even explained the contradiction.
Let's dissect this:
> No, its lots of events with the same methodology (within an election cycle, within any class of election events.)
So if that were accurate, it would require that he have an algorithm which is exactly the same from election to election, and requires no human tweaking for each one. Is that the case? Does his methodology never change, which would require that the inputs are always done the same way, too?
I think you see the problem here. The "inputs" are the polls which get done mostly by third-party polling firms, using methods that are always evolving.
Secondly, I don't believe 538 has an algorithm that's free of human intervention and never changes. What that means is, each election is a unique event.
Even if each election cycle was a unique event because of model tweaks between cycles, there are hundreds, perhaps thousands, of events and several orders of magnitude more of predictions at different times in each cycle.
No, there are not "hundreds, perhaps thousands, of events and several orders of magnitude"
There is one event. The US House, Senate, and state elections have completely different methodologies and no bearing on the Presidential election, which is enormously complicated by the Electoral College.
> The US House, Senate, and state elections have completely different methodologies
No, they don’t.
> and no bearing on the Presidential election
Even when considering the Presidential general election in isolation, there are currently 56 (51 statewide, including D.C., plus by-district elections for 2 electors in Maine and 3 in Nebraska) elections of presidential electors, which not only “have a bearing on” but strictly determine (absent irregularities that 538 doesn’t purport to address) the Presidential election.
OK, you're right, there are 56 elections, and what I meant by "no bearing" is, there has to be polling in most or all of the swing states. Furthermore, the swing states have weights. So it's a much more difficult problem than a single district, and a much noisier estimate since the per-state noise compounds.
I don't blame anyone for getting it wrong. It's a hard problem.
> Can you run the 2016 election 1,000 times and count the times Hillary wins?
I assume you mean "in real life", but otherwise this seems to be close to their actual methodology. You can see the outcome frequency charts under "What to expect from the Electoral College."[1]
Well yes, I mean, all simulations are, right? You put as much real-life data in as you can, but everything's an abstraction unless you're the team from the Devs tv series.
The actual election that happened is not a simulation, at least not the same kind of simulation that was run by Nate.
That qualitative difference is fundamental. It suffers from the same problem when results of simulations are applied to the "real world". The quality of the simulation is paramount, and there's no way to determine whether the simulation is good enough besides assertions that the simulations are really sophisticated and Nate is very smart.
If you apply the same statistically model to many discrete events, you can evaluate its accuracy. Intuitively, High confidence + Correct answer > Low confidence correct answer > low confidence wrong answer > high confidence wrong answer.
Burning my roulette wheel after winning doesn't invalidate the fact that there was a 1/38 chance of doing so.
You can stop there. Does 538 have a single model that never changes and never receives human tweaking? Does it in turn rely on third-party polls that also never change their methodology? No, of course it doesn't.
The last sentence illustrates why people's concept of probability is so off. Cards, dice, and roulette wheels are not a model for human events.
>Can you run the 2016 election 1,000 times and count the times Hillary wins? Of course you can't.
If you think that evaluating their model based on binary outcomes is the best way to do it, then, well, this argument makes sense.
If you want to evaluate the model based on all of the details available, then, well, it stops making nearly as much sense. What explanations were given for potential Trump victories? What information was available at the time? What information was available after? Do we have data around if those explanations did align with Trump's victory?
I've not done a deep dive on this - 538 is entertainment for me. But from what I remembered in 2016, what I read today in digging up some example articles for another comment, and looking at the actual outcomes... yes, those explanations were closely aligned with reality.
Humans have to come up with probabilities and make decisions based off of them all the time, even when we will never get a chance to get 1000 occurrences. These might not be provable in the same way as something we can run 1000 times is, but the idea that they are meaningless and without value is bizarre.
it's not surprising that so many people have trouble with this. Everyone studies card games, dice, and pulling a red ball out of a jar with N red balls and M white balls, because those are idealized problems.
What? At the time, people were lambasting him for saying Trump had a 30% chance, and I know people who think he was too smug after getting 2016 right. I mean, I know this is the internet and people can say whatever they want, but come on.
These are the people who got their feelings hurt over a culture war and lash out at Silver because he's become vaguely mainstream synonymous with political projections. Nobody with a serious position and serious attention on this would criticize Silver. It was the Buzzfeed types that gave Clinton a much higher chance.
No, literally everyone gave Clinton a much higher chance. I think FiveThirtyEight is mostly a clickbait circus, but they were the only serious prediction organization who rated Trump's chances as high as they did.
FiveThirtyEight had about a 1/3 chance of Trump winning, far higher than anyone else was modeling. And he was one of the few pundits openly talking about how it wasn't some insane longshot. I have no idea where this meme came from, but it's annoyingly persistent.
I remember this. Tough spot for NS. I remember thinking that 2:1 Hillary and three language around it, came across a useless hedge. In retrospect, I think he'd have gotten more respect if he shouted from the rafters that Hillary doesn't have a lock and Trump could really win. Then, he'd have been heralded as a genius. Strong forecasting, weak marketing.
For the individual prediction it's impossible to say, so you reward the people who got it right based on how likely they said it was, and you penalize the people who got it wrong based on how unlikely they said it was. If we apply a proper scoring rule, over time the you score better by giving better calibrated guesses.
I agree with this. But the original statement is poorly phrased, that "He was right." He was perhaps more right than others who gave Trump lower odds. But there's not an absolute right/wrong (unless someone gave 100:0 odds).
He seems to have been right that Trump's odds were higher because prediction error for different states wasn't independent as some other models were assuming.
I agree that we don't have a way of assessing whether the percentage given was "correct", but over a bunch of predictions we can keep track and see how he does when he gives various percentages, or compare against others with a proper scoring rule.
I don't make any claim that the odds he gave were right, but just that if they were we should still expect to see his "prediction" be wrong pretty often, so the fact that his prediction was wrong this time only counts so much against him (and to make up for that, only counts so much in is favor if he's right with low confidence).
You’re being downvoted by people who don’t have a good argument against you
To them Silver could have said Trump had a 1% chance of winning, and they would still be saying “yOu dOnT uNdErStaNd stAtIstIcS!!1”
Basically, a lot of people really really really wanted to believe Hillary winning was a sure thing. They ignored any and all data saying “maybe she won’t”
And now you can see the effect looking back - these people are still so salty about that bitter loss that they come in here and say that giving Trump a <30% chance of winning was absolutely the right call and if you don’t see that then you’re an idiot who can’t grasp statistics
Literally - half the comments responding to me was this vitriolic idiocy. They can’t think, they’re emotions have overridden that
That’s why they downvoted you for accurately pointing out that there’s no such thing as knowing that a statistical guess was “right”
They’re downvoting you because you upset them by making them accept reality. That they were wrong and the Hillary couldn’t win.
I remember leading up to Election Day his predictions were giving her 66% and Trump 33% (IIRC). Which means if you roll a die and it comes up as 1 or 2 then Trump wins (which isn't all that unlikely)
Many people mistook these numbers to mean Clinton was going to get 66% of the vote and thus were expecting a landslide, but that is not what his numbers meant.
Now I wasn't reading any of the analysis he or his team was putting out, but the modeling seemed reasonable to me.
I see this repeated all the time and I don't understand how people come to this conclusion.
538 consistently gave Trump a better chance than much of the mainstream media, with him showing a roughly 30% chance for much of the year, including right up at the election. At times it dropped as low as 15% or so, but... that's still not some insanely rare occurrence.
As for what they were actually saying? Silver and 538 never claimed it was a sure thing. Let's look at some actual articles:
"At the same time, it shouldn’t be hard to see how Clinton could lose. She’s up by about 3 percentage points nationally, and 3-point polling errors happen fairly often, including in the last two federal elections. Obama beat his polls by about 3 points in 2012, whereas Republicans beat their polls by 3 to 4 points in the 2014 midterms. If such an error were to favor Clinton, she could win in a borderline landslide. If the error favored Trump, however, she’d be in a dicey position, because the error is highly correlated across states.
There’s also reason to think a polling error is more likely than usual this year, because of the high number of undecided voters. In national polls, Clinton averages about 45 percent of the vote and Trump 42 percent; by comparison, Obama led Mitt Romney roughly 49-48 in national polls at the end of the 2012 campaign. That contributes significantly to uncertainty, since neither candidate has enough votes yet to have the election in the bag.
To be honest, I’m kind of confused as to why people think it’s heretical for our model to give Trump a 1-in-3 chance — which does make him a fairly significant underdog, after all. There are a lot of ways to build models, and there are lots of factors that a model based on public polling, like ours, doesn’t consider.3 But the public polls — specifically including the highest-quality public polls — show a tight race in which turnout and late-deciding voters will determine the difference between a clear Clinton win, a narrow Clinton win and Trump finding his way to 270 electoral votes."
https://fivethirtyeight.com/features/clinton-probably-finish... is about as bullish on Clinton as I can find from the archives, and even there Silver does not talk about how Hillary is a sure thing, and instead spend several paragraphs discussing reasons why Trump could still win and what those would look like. And as those things occurred, the model shifted back from being as bullish on Hillary, for the reasons Silver outlines.
I don't know why you would read a 70% chance of victory as being the same thing as a sure victory. I wouldn't want to bet my life on a 70% success rate!
I question this view. Why? He forecast, and whats more provided the error bounds which gave Trump a chance to win. Outside of a crystal ball, what else could be expected?
I think because what's the point of following the projections if at the end of the day they're indistinguishable from a coin toss? This applies to all polling. It's vaguely interesting but when (in Trump's case) a candidate can be consistently far behind for months and then still win because ¯\_(ツ)_/¯ that's how probability works, then what's the point of having followed the polls?
What? I'm struggling to find an analogy to help you understand what the projections actually mean, but also I'm asking "Do I care if this person is wrong on the internet?".
Maybe you need to compare it to a weather prediction. If you hear "30% chance of snow" where you're about to vacation, would you pack a winter jacket? What if it's 85%? If it's 0.5%? And if it's at 25-30% for months on end, that doesn't mean that "No chance it'll snow!".
It's not like a sports league where one person is behind for months and then suddenly wins it. Indeed what is the point of following the polls? For one thing people were probably addicted to the rollercoaster of "Oh shit my side's losing! " and then "Oh yeah my side's gaining!". At least for the campaigns themselves they should've been able to adjust their work in hopes of moving the needle (e.g. leaking stuff about the other side).
No, I understand what the probabilistic prediction means. The problem is that there is no way to tell if 538's prediction was any more accurate than if they'd said 50-50 (or any numbers whatsoever) because the presidential election is only every 4 years. We can't measure the predictive ability of his model at all. If Hillary won, we'd say "Yup, the model gave her 70%". When Trump won, we said "Yup, the model gave him 30%".
I get where you're coming from, but it's not strictly true. For instance if their model had given Hillary a 99.5% chance and Trump only 0.5%, then we could be pretty sure it wasn't a good model when Trump won.
You might say even then, how do we know the result didn't just fall into that 0.5% remaining chance? But polls can and do still give some useful idea of what the result might be.
In this case the candidates were very close. Fivethirtyeight's estimate had Hillary at 48.5% of the vote and Trump at 44.9% (and the rest on other canditates), only 3.6% apart.[1]
Usually, when things aren't quite so close, polls are a more useful gauge since they're more likely to be correct about who'll win as the gap widens.
Why are you looking at the %vote, though? In that very link, 538 has Hillary as 71% chance of winning. That is not close at all.
I'm not arguing that something can ever be treated as a certainty, just that there's "no point" in following the polls since even a lopsided 71-28% prediction will go either way, and is indistinguishable from a 50-50 prediction on election night.
What I mean is, the reason it's 71%-28% is that the actual % difference between predicted votes is very small. Once the estimated votes are say 10% apart, then you start getting a prediction that's more like 99%-1%, and the polls are much more likely to be telling you accurately who will actually win.
In this case the prediction is pretty close to just a 50/50 coin flip, but that's quite unusual and reflective of the chance legitimately being close to 50/50. Therefore I wouldn't say it invalidates polls in general.
The whole point of a poll/stat prediction from the intuitive human standpoint is to get a good gauge on how likely outcomes are
Silver, as well as all other mainstream polls/predictions in the race had Hillary winning by “the greatest margin in presidential history”
You can try and whitewash it but we all remember that. We all remember how hopeless trump looked and how bulletproof hillary’s victory was
To say Silver and all the other polls weren’t framing this as a landslide victory for hillary and a humiliating defeat for trump is just living in a false alternate reality in one’s mind.
And judging by these responses that seems to be the more common state of mind for folks here to be in…
> Silver, as well as all other mainstream polls/predictions in the race had Hillary winning by “the greatest margin in presidential history.” You can try and whitewash it but we all remember that
We don't need to remember though; we have the Internet Archive.
We can look back at Silver's prediction page on the morning of the election[1], where it's predicting Hillary winning by only 3.5% of the vote.
Yea, cherry picking is an excellent way to claim you’re right when reality doesn’t agree
Go back and get an aggregate sense of everything he said. Look at 90%+ if the articles he wrote saying what I said compared to the one or two that y’all want to keep referring to
For people discussing stats there’s a very strong inability to grok averages here…
They're all about how it's a very close, uncertain race. You might point out that the top one is "Clinton Gains...", but even that one is just talking about "Clinton’s projected margin of victory in the popular vote has increased to 3.5 percent from 2.9 percent."
I did try looking up your quote - "greatest margin in presidential history" - but the only Google result was your comment.
I was there watching. Maybe Silver doesn’t understand how to communicate verbally, only through numbers designed for consumption by statisticians. The impression he gave in his comments, if not these exact words — Hillary in a landslide.
Communication is a skill. Like the skill of a great sportscaster like Scully. Election reporting is, I’m sure, a similar unique skill set. Silver doesn’t have it, and I never want to listen to him again.
