I suppose that I could, in theory, watch it. It's a physical ballot, so we could do that. But does anyone actually do this? And more importantly, are all the votes watched?
It's completely impractical to try to watch millions of physical ballots. Or to even simply count to a million without making a mistake.
But if the voting was done (correctly, securely) in software, each person could independently audit the complete results. And to be clear, it's the overall election being audited that's important, not my vote in particular.
Perhaps it would be more difficult to understand the algorithm and code than following a physical ballot, but not a million times more difficult.
paper voting is so great because it scales so well.
There is no one person counting millions of votes. There are people counting hundreds or even thousands of votes in every precinct. Then from there they are publicly announced, and then anyone and everyone can tally up the totals.
The more people that vote, the more precincts there are, so the amount that any one person or place is responsable for (even only partially) is kept small so that mistakes or "bad actors" can only do a very small amount of damage on their own.
And if you read down in this submission a bit, people are pointing out that some countries (namely Switzerland) not only encourage people to watch ballot boxes and count along, but actually "draft" people to do it similar to a jury-duty kind of thing in the US.
Also I think you are severely underestimating the difficulty of auditing code. For starters, it completely removes the ability for the vast majority of the population to audit even if they wanted to. non-programmers can't read programming. Not to mention the illiterate, the elderly, and anyone else that doesn't have the knowledge to be able to read and follow a programming language.
Then there's the amount of time it takes. Watching a ballot box takes the duration of the election day. A day at most. Auditing every inch of a codebase? Months, years, even longer. You can point out that not everyone has to review every single line, but then you are back to letting a fraction of a percent of the population validate the code (and in a ballot box system, you can have everyone validate the process, adding magnitudes more redundancy that something shady will be caught). And what percentage of programmers do you think have a good enough grasp on cryptographic protocols and voting systems to be able to correctly and securely vet a system like this? I'd be willing to bet good money there are less than a thousand of them in the united states, and by sheer (un)luck at least one of them will be a "bad actor".
But probably the biggest thing is that to sway a "paper and pencil" election, you need materials, you need a LOT of people, and all it takes is one person to see what's going on and it all collapses. If you're really clever, you might be able to get away with changing the outcome of a single precinct, but what about the rest of the county? the state? the nation? In 2012 there were 2712 voting precincts in virginia alone. Orchestrating a system where you can put a person at a majority of them that will do something nefarious to sway the election means you are looking at 1000 people spread across a not-insignificant amount of land. And that's just for one state.
With an electronic voting system, one bug is all it takes. One vulnerability, one crypto flaw, one guy somewhere that can put the backdoor in the manufacturing of the CPUs for the vote counters. A single person can (if the are very smart and lucky) completely change the outcome of the election. And they might get caught, but the chances are much smaller than if it required thousands of people at the least to do so.
It's completely impractical to try to watch millions of physical ballots. Or to even simply count to a million without making a mistake.
But if the voting was done (correctly, securely) in software, each person could independently audit the complete results. And to be clear, it's the overall election being audited that's important, not my vote in particular.
Perhaps it would be more difficult to understand the algorithm and code than following a physical ballot, but not a million times more difficult.