Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: How do news networks get election data?
241 points by source99 on Nov 9, 2016 | hide | past | favorite | 56 comments
Seems like they have a live data feed...



AP and Reuters pay people to check the results locally. Between them, they feed a LOT of outlets. A significant amount of the data is available online pretty quickly, but some states and localities simply don't publish the data in a timely manner.

Some news organizations will have their own staff checking results, especially in areas where it is known that the results will be slow and the results are going to be within polling error margins.

You'd think the data could be crowd-sourced more effectively, but private citizens get the data more slowly for a few reasons ... First is that there is red tape involved in obtaining the data (i.e. forms to fill out, fees to pay and it all must be done ahead of time) and second is that after-hours early access to data is just plain limited logistically. If it could be efficiently delivered to a large quantity of people, it would be presented online.

There are a few areas where the government decides that it's more efficient to let private parties distribute data, and it's generally pretty good business to become one of those parties. NMVTIS data comes to mind immediately (carfax and it's competitors), but there are many similar instances.


I did this for the AP in high school. They pay you about 40 bucks and two of you go and write it down independently. You than each call the AP and state your numbers and that's it. You show up about 30 minutes before the polls close and you leave about 30 minutes after they end, pretty good pay for an hour (particularly if you are in high school)


wouldn't this be easy to "rig"?


Why would you? The numbers have no effect on the outcome of the election, just on the early reporting of it.


Because of the time zones in the United States and the electoral college system, if certain states on the east coast are called for Candidate B and it is then concluded that Candidate A has no path to victory, that might depress turnout in the western states who are still voting.


stop letting the media dictate your actions


Reuters experimented with this the last midterm, but did not for this general election. The AP is generally the source most major news outlets use in the US.

It's a very difficult operation, with state-by-state rules and some precincts that don't exactly want journalists hanging around all night.

Disclaimer: Work for Reuters News.


In my state, you go to the county board of elections website and get data about 10 minutes after its run.

For realtime data, I think the political people and press show up in person. I went with my cousin once. Some clerk taped a sheet on the wall.


DecisionDeskHQ.com, the group that powered BuzzFeed's results last night, uses crowdsourcing to gather election results.


Can a layman get their dataset in a form other than their website? I can't find info about it and their site isn't functioning extremely well.


Noticed that AP called the race well before other outlets


Associated Press offers a data feed http://ap.org/products-services/elections/FAQs https://developer.ap.org/ap-elections-api

"Shortly before the polls close, over 4,000 stringers report to county election centers. When the first polls close, they’ll be ready to start phoning in the raw vote as it is reported by the counties. They’ll place their calls to AP election centers around the country.

At the centers, a total of over 800 vote entry clerks will answer those calls, and walk each stringer through a dialogue as they enter the number of precincts reporting and the candidates’ votes into our election night system. "


But it seems not every news network uses AP data. Google and The Guardian use this data, but for instance some German news channels use another data source.

I wrote a script to retrieve the live vote data from Google (i.e. AP) for my chat bot, because I hadn't had the time to find out how the AP API works.


The AP's feed isn't free (please remember to pay journalists). News orgs like the NYT, NPR and others pay the AP a decent amount of money to manually go out and collect the results.

The AP has writeups of their process too:

http://www.ap.org/products-services/elections/FAQs

http://www.ap.org/products-services/elections/calling-races


For science, would you svare your code? #wanttolearn


Sure. It was quickly written in TypeScript nine hours ago. Google provides updates of the sub-DOM using JSON via AJAX. They use CSS class names like `._Lzm` and `._pRo`; similar to those in the HTML version of Google Mail.

http://pastebin.com/50fS83qG

Sample output (a few hours ago):

[["electoral vote","Trump 150 (120 needed) — Clinton 109 (161 needed)"],["popu lar vote","Trump 49% (35,302,284) — Clinton 47% (33,652,539)"],["progress","47 % reporting"]]

Sample output (now; notice that needed electoral votes is changed to total popular votes):

[["electoral vote","Trump 276 (57,246,646 votes) — Clinton 218 (56,294,331 votes)"],["popular vote","Trump 48% (57,246,646) — Clinton 47% (56,294,331)"],["progress","88% reporting" ]]


I did this in the 2014 election cycle, reporting a county in Ohio.

They pay $50 for someone to go to the county and report the election results. There's an iphone, android, and mobile web site, as well as a call center that takes that input.

Honestly, it was a really fun evening in the middle of nowhere Ohio...


Curious about the logistics of how you obtained the results from the county. Was it posted somewhere? Given verbally from an election official?


I hung out at the Board of Elections office. It was printed, and posted, every 15 minutes or so, as polling locations came in.

When I was there, it was me and a college student from the local college who was there working for the AP awaiting the results; everyone else was getting them online.


My county allows anyone who wants to be there to sit in a room (next to where physical ballots are counted and receipts are tabulated) and get the counts a few minutes before the website is updated.

I was there in 2008 and '12 and there are usually representatives from both state parties, both county parties, and the larger campaigns.


In Philadelphia, the city government publishes election results to phillyelectionresults.com as they're counted. The local civic hacking group (code for philly) built a nodejs scraper of the site (and an API for it) and a mobile-friendly front-end that auto-refreshes. It was available at whowonphilly.com, but the city government office that oversees elections has since adopted it as the official live results site.

https://codeforphilly.org/blog/vote_and_watch_whowonphilly

Disclaimer: I work in philly's city government. It's really cool, and we'll soon be hiring a product manager (for beta.phila.gov), a data engineer (for open data), and a front-end/wordpress developer.


The NYT had their code on GitHub: https://github.com/newsdev/elex-loader


Does anyone know if this data eventually becomes a dataset somewhere uploaded possibly free? It seems like AP's stream is for live data. I'd like the full break down by demographics and counties and stuff but clearly it's too late for it to be live.


They do -- in the form of staff on the ground collecting the information from the individual county-level offices. They collect that data as it's announced before it's even reported up to state election bodies.


The other one is Reuters. AP and Reuters are the two largest networks for gathering news. Most other brands that you know and recognize are in the business of distributing news.


I'm curious about this myself but more so on the voter registration side. For example, how do they make sure each voter is casting a single vote?

In Illinois I registered to vote well before the deadline. I showed up to cast my ballot but my name wasn't in the "database." The folks managing the polling station had to manually re-enter all my details into an Android tablet. While this was happening, I took out my phone and scanned the WiFi network of the church I was in. I assume the tablets were connected via wifi. I saw no other connection to the tablet besides a power connection. To my surprised the WIFI was running WEP. Hmm, this day in age you would think WEP would be default=off. This was at a local Church too. So perhaps the tablets use cellular data plans?

They get all my PII data entered, I get my ballot, fill it out, and pass it through a machine. The machine is in the corner of the building in a large box so I can't tell if it's hardwired to some network or using the wifi.

Later that day (about an hour later after re-entering my details into the Android tablet) I went to the Illinois voter registration web site to look up my name and I can find my details.

Anyone have any information on the tablet software? Who writes it? How it's transmitted and stored? What about the electronic ballot counting machines? Are the phoning home some where?

The whole setup seems sketchy to me.

As an aside: I know a couple people who have homes in different states and claim they can cast multiple votes by driving/flying to the state where they have 2nd home to cast a 2nd ballot.


I have some experience of this in the UK at least. Here we have feeds from the likes of the press association as well as the official results coming from the electoral commission. However, networks may have their own people at some or all of the counts. These would be local journalists who are attending the counts and will feed news stories back to the studio during the night. They will also get the results from the returning officer and call them in.

In the UK at least we aren't meant to release the results until the returning officer reads them out so waiting for any of the above while showing a live feed of a result would mean we don't have the result to show on screen immediately. For this reason you would likely also have people in the studio watching the live feeds from counts and entering the numbers which would then be double checked against the official feed later. This can be tricky when the result is drowned out by cheering from a crowd of supporters! ie "Labout party, John Candidate 22 thousand... <WOOOOOO - YEAAAH> ...hundred and 1 votes"

The focus for news orgs is getting these results out accurately before their competitors, no one wants to be slow to announce the results.


At https://democracyclub.org.uk/, for the UK, we ran live election results for our open data feeds on our YourNextMP project. That was basically just someone sitting in front of the TV putting in results as they came in :)


Back in the day it was via the News Election Service, which was a joint venture of the major news networks and AP. These volunteers (I was one of them) would go down to the voting precinct and once the votes were counted, the election officials would announce the totals for each candidate. They would write down and then phone in these results to a central office. There was a computer automated system at the other end that would ingest the results.


Well, the US seems both more advanced and more ass-backwards compared to here in Turkey. Our way of getting data is very archaic, with people checking the results locally etc. (Internet might or might not be internally involved in that). But the trust in the elections are very low, and so the coverage of the elections has been a BIG issue since 2014 or so, many news networks have a few sources in parallel, and had the numbers for each reporting outlet and their biases on screen at the same time (different outlets converge in the end, but the intermediate numbers they report can be VERY divergent)

In addition to all those, there is Oy ve Otesi, a non-profit does the entire thing with only volunteer work. Their coverage is pretty minimal in rural areas, though.


The state of Virginia has a json feed. But several times I saw news results reporting more votes than the state's website did. So it's at least not just that, if that helps.


In MN the Secretary of State makes data files (mostly csv) available to media outlets.

We transfer and process them, for national races we we the AP Election API


I don't know why there isn't a live data feed. I wish when you went in to vote there was a big screen that shows the current vote count for each candidate. You should be able to stay and watch the screen until the polls close and know exactly how your town voted.


An interesting issue most won't think if is a concept I've coined as "bandwagon voting". Essentially, if the voters can see what the current trend is, they'll vote with it. This could potentially sway elections towards one direction or the other.


Trust me, people know about this.

This was a big issue in Florida during the 2000 election. The panhandle of Florida is in the Central Time Zone and polls are open 1 hour later than other parts. The state was called for Al Gore while the polls were still open in the more "Red" panhandle area [0] and there were rumors is skewed the results or caused people to not go out and vote since the state had already been decided. [1]

[0] http://www.nytimes.com/2000/11/08/us/the-2000-elections-the-... [1] [1]


Maybe, I am not so sure that this is an actual phenomenon or just a perceived one. I, for one, think the pro's outweigh any cons, I like the the idea of being reassured that my vote got counted correctly and that there isn't any fuckery going on. I would like to have actually seen the number go up by one after I put my ballot in the machine. I can only speak for myself and say that "bandwagon voting" would have no affect on me (I voted for Johnson/Weld knowing full well they would never win).


Also, it would be pretty easy to figure out who voted for who when they step out of the polling booth and the tally updates.

You could delay it a little but live updates is kind of a bad idea.

Minutely or hourly updates might be kind of cool but yeah bandwagoning would be an issue.


>it would be pretty easy to figure out who voted for who when they step out of the polling booth and the tally updates.

I see this as a pro not a con. I want to be reassured that my vote was counted correctly. That is way more important to me than keeping my vote secret from anyone in the room.



Usually the state board of elections reports directly to the media.


Btw, Twitter definitely has the potential to rival and I think, dominate, AP and Reuters if they design the business that way. Just need a reliable way to decide trust worthiness of the tweet/source, and a way to put context around a tweet or a bunch of tweets so the headline can be derived from the tweets.


Decision Desk > NEP/NEAT for us


They get the data they are told to regurgitate from the same centralized authority that produces all other mainstream news... though they are pretty good at making it seem like true journalistic endeavors actually produce the information theyre droning out to the masses of TV zombies.


State and county websites, I think?


My impression is that pollsters, people surveying voters during the election, report via the AP or directly to networks; seems basically like the traditional news wire.


> My impression is that pollsters, people surveying voters during the election, report via the AP or directly to networks; seems basically like the traditional news wire.

Sounds like you're thinking of the exit polls, but I think OP was asking about the actual vote counts as they're reported from individual precincts.


Yeah, but the OP was also asking about demographics.


Ah yes I think you're right.


I previously read an article about this, too bad it is in Chinese http://www.wenxuecity.com/news/2016/11/07/5747980.html

basically it says it is based on exit poll. and it is costly, so many media companies formed an entity called NationalElection Pool to do the report.

And they also hired a company called Edison Research to do the exit poll.



Exit polling is different than the results the OP was asking about. The exit polls last night were wrong, they were predicting a Clinton win.

See others for what actually happens. There's reporters sitting at county clerk offices.


Yeah, but the OP was also asking about demographics. The county clerks are not the source of that information because it is not collected on the ballots. The only source of that information is exit polls.


Btw, Twitter definitely has the potential to rival and I think, dominate, AP and Reuters if they design the business that way. Just need a reliable way to decide trust worthiness of the tweet/source, and a way to put context around a tweet or a bunch of tweets so the headline can be derived from the tweets.


Last night showed all the naysayers just how powerful Twitter is as a platform. Markets are waking up to it today.

Twitter really needs to go ahead and embrace that it doesn't need to be facebook: it's key forte is being an easy to use ultra-fast realtime information distribution system.


Could you elaborate on how last night showed the power of Twitter? I haven't followed that aspect of things. Please tell.


Trump won the election after nearly a year of using Twitter extensively to reach his audience in a way that no other brand or celebrity has ever done. Twitter doesn't have a massive audience but it appears to be extremely influential and many have started to acknowledge just how influential his tweets were on voters in this election (even if they were not Twitter users themselves).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: