I think what a lot of people who don't do software development don't understand is that the majority of software is developed like this:
- Weeks, months, or years of being incomplete and full of bugs. Gaps are filled and bugs fixed continuously, but new bugs are being introduced all the time.
- At the end -- where the end is a predetermined date and not based on the state of the software -- the software is deemed good enough and put into production
So most software is just barely working, at best, and full of gaps, issues, and bugs, both known and unknown.
This scandal is a case of egregious lack of leadership, both at the post office and at Fujitsu.
But it was also enabled by a belief in the reliability and credibility of IT systems, which is profoundly undeserved and inaccurate.
Unfortunately that post claims to be the first of a series, but the series doesn't (yet) seem to continue.
It's easy to see both sides of the decision to presume or not presume that machines work - we should like somehow to explain that on the one hand it's unlikely that without his intent Jim's iPhone sent Gemma a series of messages threatening to rape her, and so we shouldn't need to drag Apple engineers into court to convict Jim - but then on other hand it's far from certain that this bank database which says I made 590 cash withdrawals in sixteen minutes is correct, more likely a bug duplicated a single withdrawal I made, and we should require the bank to produce evidence otherwise if they insist I made these withdrawals.
I can imagine that smart lawyers motivated by some real world examples and assisted by independent software engineers could come up with a general formula for courts, something a bit larger than either the old rule or the new one, but still relatively pithy. Whether a government would ever make time to do that work is another question.
This interpretation is excessively charitable regarding what really happen here...
Fundamentally, the scandal is not merely a result of misplaced trust in an IT system. It exemplifies a series of injustices, overlooked for over 15 years by complacent politicians. Crucially, it illustrates the extent of deceit possible when bonuses for senior management and technical staff are at stake.
The UK legal system's appalling quality are starkly highlighted once again. Hundreds of postmasters faced wrongful convictions and prosecutions. Individuals were imprisoned for years, without any concrete evidence of their guilt. A single miscarriage of justice is concerning, but when there are over 250, it raises serious questions about the competence of the judiciary.
Astonishingly, more than 700 postmasters were prosecuted without any executive, manager, or statistician recognizing this glaring anomaly.
The longevity of the issue is also troubling. Even low-level Fujitsu employees, aware of the system's flaws, remained silent. Despite widespread media coverage and a string of wrongful convictions, leading even to suicides, not a single person within a large team exhibited any kind of moral responsibility or something that looked like a conscience. Their silence, just makes you believe again in conspiracy theories, and raises concerns about the integrity of modern IT systems and AI that we rely on.
The simple fact that somebody decided to attest to the absence on bugs on an IT system...An achievement unheard of in the history of software development, and the fact they decided to reiterate that under oath, at a court of law, should be enough for their names to be published and banished for ever, from any profession related to technology.
> Astonishingly, more than 700 postmasters were prosecuted without any executive, manager, or statistician recognizing this glaring anomaly.
They were fully aware of the problems and deliberately not acknowledging them.
e.g. the ongoing inquiry has just been shown a document showing the Post Office indicated that it would accept a guilty plea on a less serious charge provided the defendant agreed to accept that there was “nothing wrong with Horizon”.
And to make sure the incentives were properly perverse, Post Office executives were paid bonuses for each successful conviction.
As far as I am concerned, justice will not have been done until all of their bonuses are clawed back, and at least one of the execs has been smeared on tracks by the tube.
> The longevity of the issue is also troubling. Even low-level Fujitsu employees, aware of the system's flaws, remained silent. Despite widespread media coverage and a string of wrongful convictions, leading even to suicides, not a single person within a large team exhibited any kind of moral responsibility or something that looked like a conscience. Their silence, just makes you believe again in conspiracy theories, and raises concerns about the integrity of modern IT systems and AI that we rely on.
>
> The simple fact that somebody decided to attest to the absence on bugs on an IT system...An achievement unheard of in the history of software development, and the fact they decided to reiterate that under oath, at a court of law, should be enough for their names to be published and banished for ever, from any profession related to technology.
Completely agreed. This is why ethics in our profession are important, and yet all too often dismissed.
>not a single person within a large team exhibited any kind of moral responsibility or something that looked like a conscience.
That's not true.
>Richard Roll is the real life Fujitsu whisteblower behind the character in Mr Bates vs The Post Office who lifted the lid on the faulty computer system.
> Astonishingly, more than 700 postmasters were prosecuted without any executive, manager, or statistician recognizing this glaring anomaly.
I was thinking about this last night. I knew there where about 1000 McDonald's in the UK, Googling it there are about 11,500 post offices. At some point you'd have thought someone would have noticed it was a bit excessive.
Apparently they thought that their staff were all always dishonest and this new computer system was finally revealing this truth, or something like that.
The other complication was that a change in the law was made to assume computer systems were correct by default due to the problems with the existing law around breathalysers and speed cameras:
"In 1997 the Law Commission published a paper which went into some detail about the use of mechanical and computer evidence in court. It seemed a little too fixated with the effective workings of speedometers, traffic lights and breathalysing devices called ‘Intoximeters.’ It concluded that the present law is ‘unsatisfactory’ because of the necessity for prosecutors to ‘prove that the computer is reliable.’"[0]
The amended law changed the burden of proof from the prosecution proving the system functioned correctly to the defence proving it didn't, without access to the systems being used to prosecute them.
> The amended law changed the burden of proof from the prosecution proving the system functioned correctly to the defence proving it didn't, without access to the systems being used to prosecute them.
It is mind boggling that someone thought it is OK to put this into law. What happened to the idea of innocent until proven guilty?
Seems obvious what happened in context: the government got tired of people “weaseling out” of speeding tickets and breathalyzers (think of the kids!!!) and wrote a law saying the machine right by default unless you had a reason to think otherwise. And of course prosecutors never rest at constructing a novel theory to win cases, so soon enough it was being used in serious cases and not speeding tickets…
(not that it’s fundamentally just with speeding tickets either really! but it's also kind of understandable, you're the 27th person the judge has heard today trying to weasel out of a speeding ticket with the same set of "but the machine could have been wrong!" excuses. And frankly the brits seem to be a lot more "pragmatic" about individual right vs societal ones... the political class there seems to respect the people there exceptionally little even by political-class standards lol)
Julian Assange has been imprisoned for seven years in the UK, for what were charges in the UK of a maximum, and that is a maximum of five years. The UK Legal system regularly releases on bail people accused of murder...yes murder. Assange, was setup in court sitting away from his lawyers, inside the glass-panelled dock of the court, like if he was some modern, Hannibal Lecter, ready to jump and eat the brains of the judge. Using nothing more than the similar psychological tactics, copied from Putin in Russian. Don't even pretend the court is impartial, just make the accused show up in court, with no communication with the lawyers, and inside a cage...
Judges in the UK have been so impartial they publicly made statements of him being a narcissist, not even pretending to even appear impartial. Nils Melzer, the United Nations special rapporteur on Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment, concluded after visiting Assange in prison is treatment was nothing more than torture...
That's a tricky law to have on the books. In general, I would agree that computer testimony (e.g. log records) is usually trustworthy, but that's with relatively simple processes such as writing information into a log or recording times and events. However, it shouldn't be applied to larger systems with more complicated functioning such as accounting software unless the results can also be backed up with detailed logs that can be verified to produce the stated result.
The fact that the Fujitsu employees had complete and un-audited access to all the remote terminals should have made the computer testimony unreliable, but of course the post office lied about that and denied that anyone had remote access.
This is a tricky one, because you could just claim that the system wrote audit trails that show you did something due to a bug. How do you prove it either way?
I think any (financial crime) case built solely on computer evidence is too weak to be prosecuted, even if that means you end up accepting some non-zero amount of financial crime. "it is better a hundred guilty persons should escape than one innocent person should suffer" as someone once said.
In the case of the sub-postmasters the Post Office, as far as I'm aware, never proved where these stolen sums supposedly went. The computer evidence was thought terminating and was the only thing (except false confessions under duress) used to secure these convictions, rather than proper investigative work.
Yeah, it seems crazy to me that they didn't have to prove where this money went. I suspect the majority of people caught stealing money in any other capacity are probably caught by the changes to their lifestyle being noticed before the financial irregularities are ever spotted.
The amounts people were prosecuted over supposedly stealing were in the tens of thousands of pounds (some made up the alleged discrepancy by paying back some of their own salary). Not something likely to radically alter the lifestyle of a small business owner (if they had been stealing they'd probably have spent it on cash payments to staff and family, gambling habits and boring stuff like mortgage repayments and savings)
It's crazy auditors didn't spot discrepancies especially with the high base rate of reported frauds and errors, but if finding the receipts for stolen money was the threshold for prosecution any remotely competent thief would be in the clear
It's hard to say exactly how much a sub-postmaster makes but from a brief search it seems to be in the range of £30k-£35k. It's not like the kind of money these people had down the back of their couch.
If there is absolutely nothing to corroborate the money going missing, no evidence presented for where or how the money was stolen then there shouldn't be enough to send someone to prison.
As the sibling comment says (max reply depth reached) the sub-postmasters in general were just about getting by. Part of why this scandal is so egregious is that these people were often just-about-managing under terms of an incredibly unfair contract with the PO.
The amounts of money involved may be small to business owners in other domains but for SPMs many were almost bankrupted by trying to replace the sums out of their own earnings as you say. This wouldn't have been a rounding error to their lifestyle, it should have been provable to any halfway decent investigator. And if not? Then they get away with it and it's the price we pay not to live in tyranny.
I thought when you called it a "tricky one" you were expressing that it might be a bad thing if it were difficult to convict someone based primarily on audit logs.
But if you don't want people to be wrongly convicted, then surely that's a good thing, right? As we know, there's no guarantee a particular audit log is correct.
> But if you don't want people to be wrongly convicted, then surely that's a good thing, right?
Think of it like a diagnostic test, like covid tests. That sort of test has 2 measures, not one (anyone who just says "Test X is 95% accurate!" is selling you something) - specificity and sensitivity. Sensitivity is the percentage of true positives it generates out of all positives, and specificity is the percentage of true negatives it generates out of all negatives.
I don't want people to be wrongly convicted, no, so I want legal tests to have a very high specificity. But I could do that easily, by just throwing every case out as not guilty. The hard bit is raising sensitivity at the same time. You can't just say "if you don't want people to be wrongly convicted", because that justifies far too many things.
> As we know, there's no guarantee a particular audit log is correct.
There's no guarantee anything is correct. Three witnesses could have colluded and someone might go to jail for it, but unless there's a reason to think they colluded, we don't assume that. That's the problem I'm talking about: how do we get a feel for software systems without assuming like the Post Office that they work, or like you that they don't work?
> The individual system was vouched for by senior people.
Well, how the Post Office commissioned the Horizon system is something that the inquiry doesn't seem to have got to yet.
From what I've seen of the inquiry (not much), Post Office managers seem to have been desperate to protect the reputation of the Horizon system, and still are. But the miscarriages of justice have now been exposed, and the costs of putting them right are now unavoidable; so the reputation of Horizon can't now be just a matter of subpostmasters.
My guess is that defects in Horizon, and damage to Horizon's reputation, will turn out to affect much more than the wronged subpostmasters. I suspect that some very large commercial contracts will turn out to be affected, and paying compensation to the subpostmasters will turn out to be a tiny component of the costs of this catastrophe. The Post Office is already warning that it may not be able to pay the compensation, and that the government will have to step up.
The Post Office is a weird mess of subsidiaries and holding companies, and has contracts with all sorts of 3rd parties. I anticipate that we are going to witness a slooow explosion of the Post Office, over like ten years. It's ludicrous, of course, that a private company should be able to be complainant, investigator and prosecutor as well as being sole custodian of the evidence (Horizon logs).
It's also ludicrous that the Post Office's lawyers, Cartwright King, should describe requests for disclosure as a "fishing expedition". Defendants shouldn't have to request disclosure of evidence that might help the defence; that is an obligation that falls on the prosecution as a matter of statute law.
I'm waiting for the inquiry to find out how the Horizon system was commissioned, and how it was tested; and who was in charge of commissioning.
Yes, but even after getting lots of reports from sub postmasters complaining about bugs, they persisted in telling their victims that they were the only one reporting any issues. They were lying and knew they were lying as they could just force the sub postmaster to make good any "losses". They were most definitely acting maliciously.
I can assume that a wider and deeper understanding of how software works would have meant that the investigations would have demanded a more thorough accounting of how the software worked, evidence provided by the system that it's calculations were correct (an audit trail) and that these questions would be asked at every juncture whenever the issue was raised to a new body or party.
As can be seen by following the story, this wasn't the case. The only evidence provided in criminal cases was essentially a witness statement saying that the system was working correctly and that was the end of it.
> The only evidence provided in criminal cases was essentially a witness statement saying that the system was working correctly and that was the end of it.
I'm not saying it wasn't. You're assuming a blind faith in all IT is the cause of this witness' belief. I'm disagreeing with you there.
I'm assuming an ignorance in IT which allows for a vote of confidence from a trusted party to go unquestioned and unsubstantiated. Not a blind faith in all IT.
Re "...The individual system was vouched for by senior people......"
The obvious mistake most likely made here is these senior people where not asked , " exactly how they formed this opinion." and what specific evidence they had.
I would have said , if they could NOT produce the evidence , that verified their statement, their is no way they could make the statement as accurate about the system accuracy.
The people must not have had experience in software development and testing.
Fr I feel like they need some explanation in this show for laymen that the fundamental issue here is not "COMPUTERS SCARY", it's that literally no competent software development team works on the assumption that they are infallible, and if they do these are the consequences. Fallibility is fine, the problem is that Fujitsu and the Post Office were so scared that admitting fallibility was somehow a massive moral failing. It's not. You just be prepared for it and then none of this would have happened. As an SE this is what I was thinking while watching the show throughout. But, as a layman you probably would never understand this nuance.
> Fallibility is fine, the problem is that Fujitsu and the Post Office were so scared that admitting fallibility was somehow a massive moral failing. It's not.
Fujitsu was a contractor; the contract was a PPI (Public-Private Investment) contract. The deal was that Fujitsu developed the system at their own cost, and ran it in exchange for a cut of each transaction.
I know something about this work; back in the 1980s, I worked on a bid for what was then called Post Office Counter-mechanization. This was probably the third attempt to automate Post Office counters; previous attempts had failed. FWIW, we lost the bid; I think it was won by ICL, the forerunner of Fujitsu. That third attempt failed, of course.
To work on this bid, I was loacked away in an office on my own for 6 months. The requirements were delivered as two metres of lever-arch files, full of paper forms; each form described an input screen, or a report, or a processing operation, or a storage requirement. 90% of these forms were blank, apart from the name of the operation. It was impossible to do a serious job when so little of the required data was supplied. And we weren't allowed to talk to the Post Office about the requirements; that would have risked the entire tender being overturned in court.
If the Fujitsu PPI contract was handled in the same way by the Post Office, and I assume it was, then it would have been impossible for Fujitsu to deliver a working system. But now we see Alex Chalk, the (current) minister in charge of the Post Office, suggesting that the compensation money should be extracted from Fujitsu. It may turn out that Fujitsu have some hard questions to answer; but for now, it looks as if this mess falls entirely on the shoulders of incompetent Post Office managers.
That makes a lot of sense thanks for sharing your experience. Sounds like the post office may have set Fujitsu up to fail. This makes a lot of sense since I doubt Fujitsu does not have the expertise on board to make something functional.
I don't think it was fear that led to them lying about the fallibility, but hubris. They knew that they had sub postmasters signing that the records were accurate and so they could get them to pay up for the "missing" amounts or face going to prison.
They knew the software had many bugs but decided to use that to bully their employees and steal money from them.
> They knew the software had many bugs but decided to use that to bully their employees and steal money from them.
I don't really see how this argument fits. In my understanding the amounts in question were in the range thousands to tens of thousands. It seems implausible that those sums of money, while life changing for the individuals involved, would ever move the needle at all for any of the companies involved. In fact the only people who made money out of it are likely the lawyers.
Seeing as the post office is publicly owned it feels like it shouldn't be trying to scam people for money. Nobody is seeing some massive profits from it anyway and their salaries will be decided by the government.
I suspect it was more to do with bullying than the profit. Make an example of a few sub postmasters and the rest will be too scared to raise a fuss or else be sent to prison.
Yeah, I can buy that. The enquiry even today was discussing how some postmasters were forced to agree not to blame Horizon in return for reduced charges. If that's the mentality then it absolutely fits.
Presumably those people potentially faced contempt of court charges if they continued to blame the system.
"Computers scary" is a perfectly valid takeaway for the general public. "Computers scary... because programmers/IT are fallible" isn't wrong but it's needless detail. The important thing is that people stop trusting computers to only output infallible truth, and "computers scary" gets that point across efficiently.
I've found "Programming Sucks"[1] is a good way to explain to non-programmers what a shit-show it is.
> Imagine joining an engineering team. You’re excited and full of ideas, probably just out of school and a world of clean, beautiful designs, awe-inspiring in their aesthetic unity of purpose, economy, and strength. You start by meeting Mary, project leader for a bridge in a major metropolitan area. Mary introduces you to Fred, after you get through the fifteen security checks installed by Dave because Dave had his sweater stolen off his desk once and Never Again. Fred only works with wood, so you ask why he’s involved because this bridge is supposed to allow rush-hour traffic full of cars full of mortal humans to cross a 200-foot drop over rapids. Don’t worry, says Mary, Fred’s going to handle the walkways. What walkways? Well Fred made a good case for walkways and they’re going to add to the bridge’s appeal. Of course, they’ll have to be built without railings, because there’s a strict no railings rule enforced by Phil, who’s not an engineer. Nobody’s sure what Phil does, but it’s definitely full of synergy and has to do with upper management, whom none of the engineers want to deal with so they just let Phil do what he wants. Sara, meanwhile, has found several hemorrhaging-edge paving techniques, and worked them all into the bridge design, so you’ll have to build around each one as the bridge progresses, since each one means different underlying support and safety concerns. Tom and Harry have been working together for years, but have an ongoing feud over whether to use metric or imperial measurements, and it’s become a case of “whoever got to that part of the design first.” This has been such a headache for the people actually screwing things together, they’ve given up and just forced, hammered, or welded their way through the day with whatever parts were handy. Also, the bridge was designed as a suspension bridge, but nobody actually knew how to build a suspension bridge, so they got halfway through it and then just added extra support columns to keep the thing standing, but they left the suspension cables because they’re still sort of holding up parts of the bridge. Nobody knows which parts, but everybody’s pretty sure they’re important parts. After the introductions are made, you are invited to come up with some new ideas, but you don’t have any because you’re a propulsion engineer and don’t know anything about bridges.
> Would you drive across this bridge? No. If it somehow got built, everybody involved would be executed. Yet some version of this dynamic wrote every single program you have ever used, banking software, websites, and a ubiquitously used program that was supposed to protect information on the internet but didn’t.
> You can’t restart the internet. Trillions of dollars depend on a rickety cobweb of unofficial agreements and “good enough for now” code with comments like “TODO: FIX THIS IT’S A REALLY DANGEROUS HACK BUT I DON’T KNOW WHAT’S WRONG” that were written ten years ago.
That's great! (If only a good writer could make all my points for me)
It's not all programs, but the ones that last (you know, the ones people actually use) get more and more like this over time, no matter how they start.
It's out-of-date though... allow me to fix it:
> ...with comments like “TODO: FIX THIS IT’S A REALLY DANGEROUS HACK BUT I DON’T KNOW WHAT’S WRONG” that were written *twenty* years ago.
The fact that it took a TV Show to make this blow up as much as it did really speaks to the state of the UK (and possibly other countries too). I remember hearing about this a few years ago, I remember being outraged by it, but what could I have done? Why did the people who are meant to be our leaders not do anything?
There are still other scandals out there that aren't getting the recognition they deserve (Infected Blood Scandal[1] being a big one but I'm sure there are others I'm not even aware of). So what can we do to ensure our politicians empower these campaigners instead of extinguishing them?
Great question. We're talking about systemic 'deliberate avoidance' with so many of these mistakes. How can we legislate to make sure there are penalties for this kind of malicious inaction? Government's also have to take a huge proportion of the blame.
> Why did the people who are meant to be our leaders not do anything?
I had the same thought. A while ago I heard about a so-called "scandal" in passing, and didn't think of it much. I recently listened to the BBC podcast series, and I felt both sand and outraged. I had me thinking, "does it take a TV drama for something like this to be recognized?". Now every politician is behind it (finally). Where were all these guardians before? If I find myself in a similar situation, should a no-body like me expect a documentary to be made?
Agree. The politician's incentives don't seem very aligned at the moment. Also, feel like there are some shocking things going on with the water companies.
> Why did the people who are meant to be our leaders not do anything?
This is by design. A representative democracy is not intended to do what the people want. The representatives know the people are ignorant, obedient, comfortable, and scared. So the people just stand there mouths agape while the representatives do (or not do) whatever they want. The only consequence comes from other representatives. No representative fears the people. They're sheep.
It's interesting seeing a TV drama (and, more widely, a great miscarriage of justice turned into a public outcry) centred around an IT issue.
This should be something for the HN community to take note, as this scandal is for Fujitsu the chickens coming home to roost for an issue that started in 1999. We as software developers, the maker of the tools and the facilitators of change and advancement have responsibilities, and those responsibilities can have wide reaching consequences.
We saw in Myanmar that not understanding the locality and not having adequate moderation can have consequences, we see with this case that absolute faith in a system can be damaging and mixed with other factors (such as the UK's obscure Draconian laws around the mail service being able to prosecute people) can be catastrophic.
So, friends, when you are approaching a deadline and you feel as though you need to ship even though you know the product is bug riddled, just take a moment to write some extra tests or to document the shortcomings more thoroughly. Someone you don't know may be depending on it.
So, friends, when asked if 1 + 1 = 472, don't say "of course it does, you filthy thieving criminal" because you don't want to admit you or someone else working on your innovative reimagined system maybe possibly made a mistake.
The software this scandal centres around was known for recording incorrect numbers. Numbers that were then relied upon to accuse postal employees of theft. blitzsr is saying that you should be prepares to speak up when your software is fucking up. Essentially a more consise way of saying what zhyl did.
In fact very few of prosecutions relied on actual numbers. The problem was that Horizon handles both POS and accounting, and would shut down the entire store unless SPMs signed off on the accounts. Then if there was a dispute, the PO helpline would tell SPMs "just accept the accounts, we'll investigate later". The investigators would then decide it was fraud, but drop the fraud charge in return for a guilty plea to false accounting - which technically did happen when the SPM first accepted the disputed figures.
So none of the criminal trials ever got as far as actually looking at the disputed figures, because the defended would plead guilty or the PO would just lie about how the system worked to "prove" false accounting. As Justice Fraser wrote in the 2019 common issues judgement:
"This means that, even for disputed items going back as far as the year 2000, this litigation is the first time that there will be any independent consideration of disputed items showing in the branch accounts for the vast majority of the Claimants."
Indeed many of the disputes were not even caused by software errors but human error on the part of the SPM. Errors that they attempted to resolve but could not due to the way PO handled disputes.
So this is far more than just a code quality problem.
If you are going to sit in court and say the system is flawless and could not possibly make a mistake, despite your extra unit tests and documentation of the shortcomings then you might as well just go home early and skip the extra work.
From the published details I've been trying to work out the architecture.
The 'server' piece seems to be based around an XML message store (hey it was the 90s dont forget, XML was the big new thing) called Riposte build by Escher Group.
There were some touchscreen EPOSS terminals for the post offices - these may also have been part of the Riposte system. Then there was a bunch of other stuff built around that and written in VB6 (which for the time was a reasonable choice unless you wanted to go Java or C++) running on NT workstations.
Syncronisations from the thousands of post offices to the central server seemed to be via scheduled programs that picked up local message files (probably xml) and sent them to the central server over ISDN lines.
Centrally there was something called the 'cash account' which pulled in those messages and (I guess?) added them to the riposte xml message database. As mentioned in the witness statements, they had no agreed data dictionary or schema for all the XML stuff so no centrally agreed scheme of how to represent all the different messages. So somewhere something was probably written to 'transalte' all these different messages into some nominally agreed standard.
I've watched some parts of [1] on youtube, and it seems the riposte system is on the machines in the post office. if it is also used in headquarters, I don't know. they have messages that have incremental sequence numbers to identify the replicates messages.
messages are put into the message store where each message has a checksum. when a message is read and gives a CRC error, someone raises a call, and they rebuild the message store. which recalculates the checksums, I assume.
CRC errors can be caused by data corruption on the counters. so this implies data is stored on the counters. when they rebuild the message store it seems they can use data from the other counter which hopefully has no corrupted message (but the former software engineers can't quite remember if this was true, at least she says so). so it seems they have redundancy between these counters. when CRC erros happen often, they'll replace the counter.
I also assume they describe the Horizon system pre-2010.
Reading the Riposte page from your Wayback Machine link it seems Eschergroup gave Fujitsu UK the hardware concept with redundant workstations, and Fujitsu extended it for the data integrity concept of their application [1] ("2 Horizon Data Integrity", page 6).
The concept looks really nice on paper, also the network communication considerations [2]. But there were bugs in Horizon like when a notebook at a post office was shutdown before a certain time, there were no day-end-markers added to the last transaction, and it seems they had no concept to let the system check this on the next day. There were bugs where the start times for transactions where missing, and these transactions were filtered in certain views however these messages eventually were transmitted.
[2] WITN00780100 - Richard Roll - Witness Statement.pdf
>[...]there were several different types of configuration, depending on the type of PO (mobile, single counter, multi counter or main PO with a separate gateway computer), but from the SPM's perspective these would have all looked the same.
>General security protocols were in place at the PO's. Secure passwords were required, which were not supposed to be shared but often were. There was a secure link from the PO counter to Fujitsu's servers —the counters would only respond to requests originating from specific telephone numbers —and all data was encrypted. Additionally, the network was completely (logically) isolated form the internet, it had its own dedicated lines[...]
Fujitsu casestudy PR document from 2004, still on their website! with a bit of high level info and bumf about the system. Has a few photos of the hardware.
Part of the problem is that techy people, or justice-minded people, love the detail of just how awful something is. I've been boring my friends to death for years over this scandal.
By comparison, this drama was only a few hours long, was aired while people were still in a holiday/relaxing mood after Xmas and New Year, and spun a compelling narrative about the people and communities affected. It had much clearer "goodies" and "baddies" (without being unfair). So it deserves credit in reigniting interest in the scandal.
Another thing that I fear people don't get: they see the name "Fujitsu" and thing "oh, Japanese people". Not in this case, no.
Fujitsu is just the current name of who owns ICL, International Computers Limited. ICL are British through and through. This entire scandal is "Fujitsu's UK subsiduary", i.e. ICL
ICL were established and got into UK Government IT procurement in the 1960s, because Harold Wilson and Tony Benn didn't want to be beholden to the Americans (IBM) for these new-fangled computer calculator thingies.
ICL would supply both the _computers_ and the _software_ to British government. It still runs mainframes for the Inland Revenue and Department of Work and Pensions.
By the 1980s, they were still running but they were using Fujitsu components and entire PC systems, and by the late 1990s, Fujitsu had acquired ICL... but ICL is still ICL (with a new name) and is still incredibly chummy with the UK government and still wins contracts for government IT procurement. None of this scandal happened in Japan. It happened in Bracknell.
Fujitsu (and to a certain extent, the Japanese government) is free to stop lending its name to incompetent & criminal scumbags if they don't want to be associated with it.
The drama was brilliant. I knew that this scandal was brewing; really, everyone in Britain that kept up with any news at all knew it was coming. The drama brought out just how much suffering was caused by the Post Office's lies; and just how much CYA bullshit the Post Office got up to.
I'm waiting to see a bunch of Post Office senior management in court on criminal charges.
> Although parts of the EPOSS code are well written, significant sections are a combination of poor technical design, bad programming and ill-though out bug fixes. The negative impact of these factors will continue and spread as long as the PINICL fixing culture continues. This is partly due to the nature/size of the bug-fixing task and partly due to the quality and professionalism of certain individuals within the team. The problem is probably best illustrated [by] examples:
> Example 1: This extract from EPOSSCore.dll has been written to reverse the sign of a number and is equivalent to the command d=-d
Public Function ReverseSign(d)
If d < 0 Then
d = Abs(d)
Else
d = d - (d * 2)
End If
ReverseSign = d
End Function
> Whoever wrote this code clearly has no understanding of elementary mathematics or the most basic rules of programming.
2. Here are the juicy details about the overall architecture:
> The Fujitsu-designed Epos software on the PCs was written onto an off-the-shelf system called Riposte.
> Our source said the big flaw in Horizon was the way data was being written to Riposte.“Riposte wasn’t really a database, it was a messaging system based on an XML structure where you write messages down into the message store, and then Riposte took care of replicating them,” he said. “The first thing that you should always do with a system like that is design and agree a data dictionary and a message library repository, basically to say: these are the messages that are allowed to be written to the message store and they all provide the following function.
> “You should also have a layer of software that lies on top of the message store that checks that any application above it which is trying to write a message, conforms to the agreed data dictionary. Otherwise, you can just write freestyle to the message store, which is what they were doing. There was no application interface in there, no agreed data catalogue or anything.”
> Computer Weekly also spoke to a former Fujitsu employee who worked in the Horizon Service Support Centre from 2001. The support engineer – who also wants to remain anonymous – recognised this new description of Horizon’s badly built message store, adding: “Our job was to fix these problems as they arose. We all knew the code wasn’t fit for purpose and needed rewriting. The data dictionary was still being added to when I got there.”
> For the first 10 years of Horizon’s existence, transaction and account data was stored on terminals in each branch before being uploaded to a central database via ISDN. Our source says this part of the system simply did not work.
> “The cash account was a piece of software that sat on the counter NT box, asleep all day,” he said. “At the end of the day, or a particular point in the day, it came to life, and it ran through the message store from the point it last finished. It started at a watermark from yesterday and combed through every transaction in the message store, up until the next watermark.
> “A lot of the messages in there were nonsense, because there was no data dictionary, there was no API that enforced message integrity. The contents of the message were freehand, you could write whatever you wanted in the code, and everybody did it differently. And then, when you came back three weeks later, you could write it differently again.”
He gave an example of a message stored previously when a customer bought a stamp. It was feasible that a new message for buying a stamp weeks later could be slightly different.
> “When the cash count came along, it found a message it was not expecting and either ignored it, tripped up, or added something it shouldn’t be adding,” he said.
> “You should also have a layer of software that lies on top of the message store that checks that any application above it which is trying to write a message, conforms to the agreed data dictionary. Otherwise, you can just write freestyle to the message store, which is what they were doing. There was no application interface in there, no agreed data catalogue or anything.”
I think we can see that this sentiment has wider support in the software community - I would put many NoSQL database adaptions into this category. Another interesting development that shares this sentiment is frontend developers desire to bypass backend layer and get direct access to the data to "easy" and "speedup" their development.
But nothing in your quotes shows anything especially incompetent - there's no smoking gun.
The world runs on badly written software. I have even written some highly incompetent code myself for essential services. Finding terrible bugs and then looking for who caused them has almost humbled me once or twice.
>But nothing in your quotes shows anything especially incompetent
They did not care about data integrity and schema migration. It's probably the most incompetent thing one can do when working with data in such a system.
That's why we define database schemas, or write protobuf schemas, or avro schemas, and parse JSON data in the frontend with zod.
As a rule of thumb you don't even just change the datatype of a field and hope it works. You introduce a new field and migrate over the course step by step from the old field to the new field, i.e. you write the data into two fields, and read the data from the new field or fallback to the old field.
Actually their use of the Riposte message system in 1999 was not that different in what we use today in many cloud-based systems with microservices.
Yes, but at the end of the day nobody wanted to lose face and admit the system was flawed and people ended up in jail/going bankrupt/committing suicide or all three because £10,000s of discrepancies were assumed to be fraud and not bugs.
Yes. Flipping the sign of a number can typically be done in one line as `d = -d`. Or if that syntax isn't available in that programming language, then `d = 0 - d` or `d = -1 * d`.
What's wrong with code above is 1) it does a conditional branch (meaning if.. then... else...) which is unneeded. 2) The "then clause" does an absolute value which is a waste of a function call. 3) The "else clause" does `d = d - (2 * d)` which is a waste of a multiplication 4) possibly with an overflow risk and/or floating point estimation risk.
That said, it's admittedly possible that the programming language above, or the number representation in that code above, have some highly atypical needs that we here on HN don't understand.
They had teams with software engineers with different levels of competence and experience [1]. There were good, mediocre, and members that were not so capable. Which is true for many teams. When at least the good members have the time and review the code of the others, than this code would probably not go into production.
Even when these code snippets are cherry-picked they tell enough about their review practices.
>have some highly atypical needs that we here on HN don't understand.
Somebody else checked in another thread. You can use unary minus in VBA.
[1]
>One member of the development team, David McDonnell, who had worked on the Epos system side of the project, told the inquiry that “of eight [people] in the development team, two were very good, another two were mediocre but we could work with them, and then there were probably three or four who just weren’t up to it and weren’t capable of producing professional code”.
it appears correct but over-complicated -- it should be a single line of code. the over-complicated solution may have underflow/overflow problems with very large numbers.
So yeah, when I was a youth around 1995 I wrote a Turbo Pascal application for my local post office in the the North of England. On the house, no payment - mainly to hang out in the back room and see the world go by. The application tallied up the days ins and outs and then printed the results on dot matrix printer which were used to help with the book keeping. The post master and mistress were very happy with it. Once they were invited to the post office headquarters in London for some award for their services and I tagged along for the trip. After some years I heard the post office shut down due to some financial scandal and the post master lost his job and was forced to become a taxi driver. Never found out the details until now. Blows my mind thinking how much money goes to people who don't actually do shit and actually hold back others trying to earn some peanuts. I remember fondly those times, especially watching the post mistress hypnotised with my built-in psychedelic screensaver. She was well impressed.
Of course, I have to take this with a grain of salt, can't verify it, but what a story.
Post office should have used your Turbo Pascal-based system, the post master would not be forced to become a taxi driver.
This reminds me I always wanted to build a fictional cashier system with terminal interface in ncurses, after I saw these still used in early 2000s in DIY stores. Maybe a local post office system would also a great side project for the weekends.
Don't focus on the IT/software stuff - it's missing the forest for the trees. Some people really do need to go to jail, and it's the people at Fujitsu, the Post Office, and the prosecutors offices who were involved and negligent. They weren't all negligent but some of them were and it needs more than a civil investigation.
Any engineer at ICL who was present when these prosecutions were going on, and *knew* there were problems with data integrity and quality of the system in general should've blown the whistle and shares (some) responsibility for these repeated miscarriages of justice just as senior management at the Post Office do.
If folks here don't share this view in their day-to-day work, then they are part of the problem. The act of writing software does not exist in a vacuum and software engineers are not too stupid to be able to consider the wider consequences of what they are doing.
In other professions, acting honourably actually matters and there are professional bodies which actually give a damn about the ethics and actions of their members. I'm not sure why it's okay for software engineering to remain in the dark ages.
I think the majority of the blame should be on the higher ups at the Post Office as they weren't so much negligent as being malicious. Fujitsu were more likely incompetent, but they weren't the ones who prosecuted the sub postmasters and perjured themselves.
There definitely needs to be some heads rolling for perverting the course of justice if nothing else. I'm surprised that the court system is happy with the Post Office using them as their attack dogs.
Currently, two somewhat lowly staff engineers of Fujitsu are being prosecuted for perjury: Gareth Jenkins and Anne Chambers.
Why specifically them? Because both gave evidence in Post Office prosecutions of SPMs that Horizon did not have flaws, then in the Bates v Post Office Ltd case, their support ticket logs (PEAKs) were finally included in evidence. These work tickets show both of them acknowledging bugs that create false accounts, and manually correcting data in the records and audit logs to balance it out... and also sometimes getting the "fix" wrong without noticing... and noting that they saw no need to inform the SPMs.
The very things SPMs said were happening and the Post Office and Fujitsu denied. They fucking knew it. They testified that the things they did day-in, day-out never happened, and SPMs were convicted and sometimes imprisoned as a result.
Obviously there are a lot more people at both Fujitsu and the Post Office that need to see justice, but this is a start.
> The Postmaster reported that the problem at his one specific site had been been happening intermittently all year. He told NBSC who sent experienced trainers out to ensure the Subpostmaster wasn’t doing something wrong. The trainers concluded the Subpostmaster was doing nothing wrong and witnessed the error happening. The auditors, who were incentivised by the Post Office to suspend Subpostmasters with discrepancies (another story), came out. They concluded the Subpostmaster was doing nothing wrong and they witnessed the error happening.
> Chambers went to work. She delved into the system and reported: “have checked the system figures… and can confirm that all the variants reported since then have been calculated correctly. There are no known issues that would result in the variance being incorrect.”
> Chambers closed the ticket with a definitive: “No fault in product”.
> The cause of the defect was assigned to “User” – that is, the Subpostmaster.
> When Beer asked why, Chambers replied: “Because I was rather frustrated by not – by feeling that I couldn’t fully get to the bottom of it. But there was no evidence for it being a system error.”
...
> Chambers conceded: “something was obviously wrong, in that the branch obviously were getting these discrepancies that they weren’t expecting, but all I could see on my side was that they were apparently declaring these differing amounts, and I certainly didn’t know of any system errors that would cause that to happen, or that would take what they were declaring and not record it correctly…. so I felt, on balance, there was just no evidence of a system error.”
> No evidence. Williams pointed out that it surely was unlikely to be a user error if both trainers and auditors had recorded the Subpostmaster as inputting information correctly. Chambers replied:
> “Well, yeah, I… yes, I don’t know why… I’m not happy with this one. But I still stand by there being no indication of a system error and the numbers that they were recording just didn’t make a lot of sense.”
The more I learn about the scandal, the more I think that everyone involved in it from the Post Office and Fujitsu should be prosecuted as taking part in a criminal conspiracy to pervert the court of justice.
Print-outs of the support system tickets are shown in the inquiry [1] (e.g. the ones with former technical support engineers). For instance, there is one from October/November 1999 with conversation history of people from different levels of support, where you can see they (the support and software engineers) knew the exact amount of money missing and that the caller was "very agitated as he keeps having problems with the system when balancing. He thinks it is a system problem".
I haven't seen support system tickets where they stated the discrepancies were caused by Horizon in the inquiry (obviously in court they proved there actually were), but I only watched a few hours of this so far. And the support engineers stated they did not grasp the implications of the callers having discrepancies.
The support engineers even visited the callers on-site.
If I go into a police station and file a police report saying you stole $50,000 from the Post Office branch you ran, one would hope that I'd also have to prove it. If I filed a civil complaint against you, I'd have to produce evidence.
In both cases, if I lied by, for example, saying that I had evidence that I did not, I could and should be charged with a crime, in the very least making a false report and/or making a false statement to the court.
Yet we let companies do this with IT systems all the time. Hertz famously accused people of stealing cars that they did not [1]. The same principle should apply: whoever OKed such a system should be charged with a crime.
All these systems make the same mistake: they want to replace humans. If you want to do that, you should be on the hook legally for the results but really it shouldn't be allowed.
It's completely fine if Hertz designed a system that flagged people for not returning cars (ie stealing them). That would get sent to a person. That person would review the information, check if it was an error or not and, if not, file a police report. That human would be responsible for maaking a false police report in the case of an error. Any errors could then be addressed as bugs with the systems provider.
Likewise, for Horizon, by all means automate the review of financial records. Find what you think are fradulent transactions. That gets spit out to a report. That report may then be investigated by, say, a financial accounting firm who is responsible for determining the facts and, if necessary, taking legal action.
You see how this version aids a person doing this job and the actual versions of these replaces the humans? A human can't necessarily review every transaction of the postal system so automating fraud detection makes sense.
What doesn't make sense is allowing the systems provider and the government agencies that enabled this off the hook because of "bugs".
Not to excuse the post office or the UK legal system-- but there are reasons your point has limitations.
The subpostmasters were signing statements that their accounts were accurate, their charges were mostly related to these sorts of collateral lies rather than the theft. So in a uselessly pedantic and literal sense they often had performed the wrongful act they were accused of, though this ignores that they were more or less forced into it and sometimes explicitly told by tech support to do so.
In specific cases the financially disabled and often unrepresented subpostmasters made legal errors. For example, instead of defending themselves by responding that the Horizon figures were inaccurate they defended themselves by saying they could prove they didn't take the funds from their own books. Essentially allowing the burden of proof to be reversed.
None of that is incompatible with what my comment.
So subpostmasters signed affidavits to the accuracy of their accounts. The burden of proof should still be on the government to substantiate that those claims are false and there is material harm and not, for example, just a bookkeeping error.
Particularly when a system is new, we all know that it has to be verified. For migrations that typically means comparing the results to the old system and then manually looking at discrepancies. In the case of a new system, it should be the bare minimum standard that before a person is accused of a crime, the system is verified, in this case by human review, probably by an auditor or financial accounting firm.
The crime being the false affidavit (even here you have the government has the burden of proof to show it was knowingly false) rather than any alleged theft is pure theater and a cop out. Like you sign a statement saying you didn't steal money and I charge you with lying. Shouldn't I have to prove you were lying? Isn't that basically the same as proving the theft occurred?
If subpostmasters can be charged and/or sued for making false statements by a computer system, why can't someone be charged for the false statements that computer system made?
I think you have it backwards, they signed statements saying the system generated false accounts were accurate. In some cases they did so after putting in manual adjustments to diminish the debts which themselves were not accurate (they had the right moral effect, but they were literally false accounting records-- correct accounting would have required finding and adjusting the actual errors made by Horizon, which they did not and likely could not do).
At best they were in a weird catch-22 where disputing the records is an admission that they knowingly filed false accounts (itself a crime, regardless of they funds owed or not), and at worst they indisputably entered false accounting records (though for understandable reasons and sometimes at the direction of tech support staff).
Part of the reason justice was possible at all was because Alan Bates simply refused to sign the false reports. He got shut down and suffered tremendous hardship as a result, but he wasn't litigated against-- which left him in a position to drive forward justice. ... and even that required whistle blowers and leaks to force the disclosure of documents that were wrongfully being concealed.
I can't emphasize enough that I'm not saying any of this to attribute fault to the subpostmasters or deflect it away from the post office or Fujitsu. I don't think it does.
But I think it's important to have a more comprehensive understanding of what went wrong so as to understand how situations like this can be prevented.
To that end understanding how the situation made some of the subpostmasters guilty (or at least in violation of their contract) in narrow a technical sense is important, because it highlights how the judiciary and its duty bound actors (the solicitors, barristers, and independent experts) operated in a -- for lack a better word-- autistic and rule bound way that yielded results which were completely incompatible with justice.
The SPMs were forced to sign and take responsibility for accounts that were prepared by the Horizon system, which claimed the SPMs owed the Post Office tens of thousands of pounds of imaginary money. If they later claimed that the accounts were wrong (which they were), they'd be prosecuted for signing them, and if they didn't do that they would be prosecuted for not paying the PO the money they supposedly owed. So they would lose either way, and the actual accuracy of the accounts was never tested in court, because in any individual case the PO did not disagree with the SPM about that.
The point being that this can't be blamed on software bugs alone. It was caused by different departments of the PO having completely contradictory ideas about how the software was supposed to work.
>Any errors could then be addressed as bugs with the systems provider.
Were bugs actually involved? I could only find this sentence in the article and it leaves it open if it was a bug or an user error.
>Many of the Hertz cases involved customers who had called to extend their rental agreement, but the extensions were not properly reflected in Hertz's computer systems.
There are hundreds of cases, all different. Many involve software bugs but at least one does not. You can read about that one in the common issues judgement paragraph 217.
My comment was about the Hertz car rental system, not Horizon.
Regarding software bugs in Horizon: Some of the bugs are mentioned in the early chapters in the book by Nick Wallis. I first read an article by the Guardian which only mentioned two of the bugs (as far as I remember) but also said many other bugs were discovered [1]. I first thought these were mostly UX issues [2] [3] that caused user input errors, but reading some of the inquiry reports and the book made it more apparent that there were system bugs, e.g. [4] on top of a sub-optimal message system [5]. Of course, you can say UX issues are also some kind of bug, error or defect.
[1]
>As early as 2001, McDonnell’s team had found “hundreds” of bugs. A full list has never been produced, but successive vindications of post office operators have revealed the sort of problems that arose.
[2]
>“The Dalmellington Bug entailed a user repeatedly hitting a key when the system froze as she was trying to acknowledge receipt of a consignment of £8,000 in cash. Unknown to her each time she struck the key she accepted responsibility for a further £8,000. The bug created a discrepancy of £24,000 for which she was held responsible.”
[3]
>"If a bunch of products sit in a basket for long enough on the screen Horizon will turn them into a sale automatically." -- ‘It is an example of Horizon doing what it was supposed to do.’ -- ‘It is evidence of the system doing something without the user choosing to do it.’ retorts Mr Coyne.
>"This stance of blaming the users if they were confused by Horizon’s design was taken to an extreme with “phantom transactions”. These were transactions generated by the system but which were recorded as if they had been made by a user"
[4] >In April 2003, Horizon version S30 was released by Fujitsu. It had not been properly tested, and it introduced a bug known as the Reversal bug. If a Subpostmaster inputted a series of transactions, and then for some reason wanted to cancel them, the transactions would have to be reversed out of Horizon. The Reversal bug appeared to reverse a transaction, but in fact doubled it. This was due to a plus sign in the code which should have been a minus sign. The Reversal bug was discovered by Fujitsu after Subpostmasters began to complain about the growing amount of non-existent money Horizon calculated as being in their accounts at the end of the week. When one Subpostmaster reversed an internal £13,910 cash transfer out of one stock unit and into another, he managed to create a £27,820 negative discrepancy in his accounts.”
[5] > "The biggest problem, however, was the information being written to something called a message store.”
>“According to Clint, Horizon’s developers had spent the last couple of years ‘firing any old shit’ into the message store as they changed and developed the product. Clint explained that this was a recipe for disaster. If a message was given a new data field, but this wasn’t retrospectively applied, the counter software could retrieve an old message with what it would now perceive to be a missing data field. The reaction of the application would be unpredictable.”
> All this has left an intriguing question: how has a TV show achieved in one week more than investigative journalists and politicians in more than a decade?
My answer is that information can easily be buried by the media, courts, politicians. The purpose of media is to guide not inform - there is nothing natural about the news.
The more interesting question in my mind, is why is this news now? It can't be that a genuine issue has filtered into the public consciousness via a dramatised TV show.
Is it that the show and surrounding dreams is intended as a distraction? Is something in the drama 'reframing' reality? Perhaps it is intended to show how 'the justice system may be slow, but still calls people to account' ie an attempt to shore up faith in our system? Is the loss of someone's CBE (an honour) intended to be a lightning rod to channel popular discontent down a cul de sac?
If the drama enables people to question whether the result their software gives them is correct, then it does a good deed indeed. The justice system being that slow is an indicator that it needs more resources.
The UK honour's system... Liz Truss' most chaotic seven weeks in UK politics and she has the impudence to even consider, let alone release the traditional honours list of a Prime Minister leaving office. https://www.bbc.com/news/uk-politics-67843755
No one cared in the US about Bill Cosby's decades of rape allegations, either, until a recording of a stand-up comedian making a joke about it went viral and encouraged people to google "Bill Cosby rape".
It's weird, I'm not sure where I first heard about it but I was well aware of the issue. The problem I had was that I just lumped it with Therac-25 and Araine Flight V88, cautionary tales from the past. I had no idea that postmasters were still awaiting justice.
It seems you have missed my point though. Your example strongly supports my thesis that this event requires a media analysis.
From your linked article:
> It was in 1951 that BBC radio and the Ministry of Agriculture created The Archers, an "everyday story of country folk", which encouraged farmers to try new techniques to increase productivity in the years after WWII.
> "It's one thing to read a persuasive argument in print it's quite another to be persuaded by the power of the human voice," says Andrew Crisell, Professor of Broadcasting Studies at the University of Sunderland in the UK.
TV shows, news, the BBC, ITV, papers, etc, etc more broadly "stuff on screens" are used to deliver a message. Media literally mediates reality for you - this is how governance occurs. One is given a message, and then most people respond appropriately. (Obv. psychological 'tricks' are known and applied, eg reverse psychology.)
The question for me is what is the message that the media is attempting to be convey in this case? Answers on a postcard.
> It can't be that a genuine issue has filtered into the public consciousness via a dramatised TV show.
Except that is exactly what has happened. The story was widely reported, nobody cared. Footballers wives fighting in court, people care. The royals, people care. Celebrities eating bugs in the jungle, people care.
If a person affected was on love island or dancing with the stars, people would have cared.
1. The public inquiry has begun digging into what actually happened.
2. An absolutely excellent TV drama has brought out in excruciating detail the misery inflicted on subpostmasters, and the unfairness of the processes.
The returned CBE is a consequence of the drama and the resulting public interest, not a cause.
I knew that this nonsense was going on, in a sort of subliminal way; I knew about the miscarriages of justice, from the MSM. The drama has - well - dramatized it. The drama's producer has expressed astonishment at the impact her show has made, but I'm not astonished. It's a really good drama. It pulls no punches. It must have been really hard to make, given that many of the people criticized are very powerful and rich, and that criticizing the rich and powerful can land you with very large bills to defend libel suits.
This event requires a rethinking of responsabilities and the power structure in the Post Office (but probably in all external goal oriented orgs). What a media analysis, in all its sloppiness, will do is make people believe there's a format for exposing scandals that can be used at will.
The point is that much was obvious for years. This particular scandal has been widely and publically reported on for years and years. Why did it take making a TV show for action to happen? That part goes way beyond the post office itself.
I don't like conspiracy theories, but the ruling Tory party jumping on to this as a blame match against Labour (for the scandal dating to their reign) and their leader (being head of CPS during some of the prosecution period) and taking swift action to legislate something to overturn and compensate the victims of this great injustice and this whole thing being timed right before an incoming election that was more or less guaranteed to be a disaster for Conservatives implies either 1) a massive conspiracy orchestrated by the greatest British minds 2) or an incredibly opportunistic move to capture the public mood.
It was reported quite a few times in the BBC, HN, and probably others, in the past couple of years. I think it had plenty of coverage but nobody really cared - it really affected business owners.
The software is dogshit as well as the company, their investigators and Post Office. But I think a lot of people were pressured into plea deal's or something, pressured/forced into repaying money they never stole, which complicated things quite heavily from a top-down analysis of these prosecutions. But the idea some random postmasters would collude in this all over the country is kind of crazy and I'm not sure how post offices are that profitable to begin with for each branch to have £40,000 stolen or whatnot..
I'm amazed, seems incompetence , starting with senior management and rippling down.
I note any person could have reported this issue , by sending a few anonymous messages to several relevant people - starting with senior people and mentioning their responsibility's and casually mentioning a few facts that only a limited number internal people likely to know etc ....
And if nothing happened "in a while" , send information to a wider circle of people
As A Software test alysist for 20 Years, working in a company that was "bank like", where numerous transactions could involve large sums of money. When software changes where ready to go to production, There would be a audit of the testing. This would take at least a few DAYSs. The (government) customer would hire independent very experienced software engineers / testers. We would have a test system that mimicked the distributed production system. with all (100s) of software versions matching . We also had internal and external auditors .
Our test plans . test cases , test data and test results would be examined. Some tests would need to be reproduced in the presence of a auditor. Or the auditor would ask for tests to be run using their provided data. We also had load testing, deployment testing, rollback tests etc. The software had to conform to certain international standards for security - so such remote access would not have been possible, in our case. If it had existed it would have been discovered through audits . Other measure s where taken in system , such as encrypted of messages and encrypted databases . Access was tightly controlled . Software developers could not deploy to prod. Lots other security controls to
I have the feeling that even a single capable software engineer from a bank (or any client) who is put into different teams of the contractor and reports back to the client, would already make a big difference. Does not replace testing at all, but would at least inform the client about possible problems, so they can stop the whole project early enough.
A worrying spin-off from this is the constitutional change. Allowing Parliament to overturn a decision by the courts is unprecedented and a worrying change.
Parliament is completely sovereign. It can overturn laws, never mind decisions based on interpreting laws. UK parliament has even made lots of ex post facto laws over the centuries. It's one of the fundamental disagreements, along with taxes, that led to the foundation of the United States.
Here's one example: Alan Turing was convicted of "gross indecency" (having gay sex) and ultimately took his life rather than live with the chemical castration ordered by the courts. It took a lot of convincing, but in 2017 Parliament ultimately passed a law pardoning all men convicted of that former crime. It doesn't matter how correctly the court interpreted the law in 1952, it was fundamentally a miscarriage of justice that we recognise in the present day.
It has already been determined by other court cases that many of the court cases that convicted SPMs were unsound, particularly because the court took the Post Office's word for it that their software worked correctly, and we know it did not. Rather than wait years to thoroughly re-examine every case on its merits, which would in itself deny justice to anyone who dies of old age in the process, we should move forward by annulling all of the Post Office's cases.
We don't have a constitution, you mean a change of legal precedent. I agree that's its and awful precedent to set. But it's already been done with Parliament declaring Rwanda safe, changing convictions doesn't seem so beyond the pale. I'm honestly not sure what else to do about it though. The sheer scale of incompetence and cover up in the post office and the software company will likely take years to sort through. Can the victims really wait that long? What is the alternative? Having everyone convicted file an appeal? After everything they've already gone through? Frankly, just wiping the slate clean and getting rid of all the convictions seem like the good move to me.
If courts interpret legislation passed by Parliament in a way that Parliament thinks is not what it intended, then the correct thing to do is to pass superceding legislation to clarify the matter. That's not a violation of the separation of powers; that's what's supposed to happen.
Given that we clearly had an exhaustive list of all of these miscarriages of justice, I too would have preferred that we fixed our absolutely shambolic and chronically underfunded court system to process the backlog properly. But given that is unlikely to happen in this or presumably the next parliament, quashing the convictions seems like an acceptable outcome. And we only just passed a law saying "Rwanda is a safe country because it is a safe country" to route around the legal system, this stuff is hardly unprecedented.
The appeal courts in the UK hate overturning decisions of lower courts; you have to have a pretty rock-solid cased to get to appeal. I'm not aware of an appeal court ever overturning a whole class of convictions.
I can imagine an appeal court ruling that some accounting system produced unreliable evidence; but each affected case would then probably have to go to appeal.
This scandal is just one of very few instances where a uk government department has been forced to backtrack, appologise, and compensate for life devastation. They literally get away with murder in departments such as DWP & HMRC.
A relevant comment from an earlier thread on this topic:
Interestingly the TV drama doesn't mention Adam Crozier, who was CEO of Royal Mail 2003-10. I wonder if this is at all related the fact that Crozier was also CEO of ITV 2010-2017, the company that made the TV drama? Hmmm. - https://news.ycombinator.com/item?id=38931500
Why isn't more software just... made good? I understand that often it's because the devs are underpaid, but is that the case everytime this happens? I get the impression (from living in this world and using a lot of software) that there are plenty of devs out there who just don't care enough about their craft to do good work, and that makes me very sad.
Such a good question. Don't blame the developers too quickly.
In most industries you need to actually make or do good stuff to have a good career. In the software industry you can (badly) manage failed projects, or lead the development of poor software products, or run internal fiefdoms that make the development of good software harder, and still have a stellar career.
Because you can make almost as much money making it badly as you can making it properly. Especially in 1998. And especially with huge government contracts.
In fact with huge government contracts in the 1990s you could make more money doing it badly than you could doing it properly. Because if you quoted the price it would have taken to do it properly, the government would buy it from your competitor who was quoting the price to do it badly.
Eventually crap project goes live once government has paid 4x the original estimate for it (because its better for everyone involved to pretend its not crap) and the company goes on to use that 'success' to bid for their next project.
For large government contracts there's no incentive to make it good, and huge numbers of countervailing pressures that make it bad. Once it's built, it's already been paid for.
The rich young ruler approached Jesus and asked him what he had to do in order to inherit eternal life. And in a perplexing response Jesus begins to answer him. "Why do you call me good? No one is good except God alone."
Software is neither good nor bad. It's run in a box of sand, and it's a minor miracle that humanity has found a way to make a box of sand do anything at all, let alone the digital empire that we have built.
Good software isn't even describable. "Well, it should be correct." Okay, well, if good software is correct software, then why didn't you use that word to begin with? But what do you mean by 'correct'? Any definition you give will have edge cases that you didn't think of. Your resolutions will self contradict. And when the whole thing is deployed we'll discover that the assumptions of what correct means do not exist in physical reality; our house of cards painstakingly constructed with insane attention to detail collapses into ash.
Jesus then goes onto to tell the rich young ruler to follow the second half of the 10 commandments. ~Yeah, I've been doing that since forever.~ One thing he lacked, though. He was to sell all his possessions and follow Jesus.
This is not a rebuke. Do not stop asking for good software. It should be your demand; it is what we all deserve. But good software isn't a checklist of rules. It is not a simple thing to ask for. It is a lifelong journey where the destination is to get further than those who have come before. Good software is a limit as our striving tends to infinity.
It's not that it runs its own court system, but the Post Office was allowed to bring private prosecutions, which is to say a private individual or organization can bring criminal law cases to court as well as vastly more common civil law cases.
What the Post Office did have, though, is its own centuries-old investigation team, and much of the scandal hinges on the fact that these investigators are employees of the Post Office and likely knew the accounting system was faulty, and those faults could explain supposed losses, but deliberately disregarded that in bullying and prosecuting subpostmasters into accepting liability for phantom losses.
Also, as well as in practice exercising control over the Post Office as its owners, the Government is responsible for the Crown Prosecutors, the people who usually carry out criminal prosecutions. For a long time now Crown Prosecutors have been able to seize any existing criminal cases (prosecuted by somebody else) in order to terminate it. They could have stepped in, taken over the case, said "This seems like bullshit" and terminated the prosecution. That's still awful for the postmasters, who have faced a threat of prosecution, but it means none of them would go to jail. Governments of the day did not do this either.
Sorry, it wasn't a "TV show" that brought this to life. Private Eye have been writing about it for 10+ years, the BBC and other papers have covered this issue multiple times in the last 3-4 years that I remember.
So it's not like the stupid people of Britain woke up one day, turned on the TV and said "Golly gee, post office scandal. I never knew!"
And this is the real reason it took so long for this program to be made. If they hadn't insisted on it being sunny, filming could have completed years ago.
- Weeks, months, or years of being incomplete and full of bugs. Gaps are filled and bugs fixed continuously, but new bugs are being introduced all the time.
- At the end -- where the end is a predetermined date and not based on the state of the software -- the software is deemed good enough and put into production
So most software is just barely working, at best, and full of gaps, issues, and bugs, both known and unknown.
This scandal is a case of egregious lack of leadership, both at the post office and at Fujitsu.
But it was also enabled by a belief in the reliability and credibility of IT systems, which is profoundly undeserved and inaccurate.