Those postmasters defending charges of theft against them in the 2000s should not have had to prove that the computer system they were forced to use had bugs. In order to prosecute them in the first place, the Post Office should have had to prove, beyond a shadow of a doubt and without risk of bias, that the computer system was correct. So, independent review at the very least, not testimony from parties with a vested interest in the outcome. This mess was as much a failure of the UK legal system as it was of the active efforts of the Post Office and Fujitsu to deceitfully protect their own interests above the postmasters affected.
It's like a murderer giving evidence against a random stranger and being believed at face value because they provided all the evidence first hand.
As the post office is such an old organisation (350 years), it used to have its own armed guard, has its own investigations branch and conducts its own prosecutions. The police wouldn't get involved because the post office was considered to have jurisdiction. Taken from this, which is a great listen: https://soundcloud.com/privateeyenews/page-94-the-private-ey...
On internal UK news, there is nothing as good as Private Eye. Every two weeks they publish more “hard” material than newspapers do in a month. I’m a subscriber, the value for money is simply ridiculously good.
I have also recently subscribed to Private Eye - mostly because there is now so little other investigative journalism going on in the UK that I think they deserve some support (the main papers are nearly all owned by billionaire mates of the Conservative Party). It is also quite funny.
They should be stripped of those powers immediately in my opinion. It's an anachronism and as been shown by this incident they are not fit for purpose.
Private prosecutions are not unusual in England, although the tide may be turning against them. For instance, the RSPCA recently announced that it would stop bringing private prosecutions for animal cruelty [0], which it has done since before there were police. There was apparently pressure from MPs for them to do so, after some fairly high-profile cases where they were seen as being too eager to prosecute.
Of course, the CPS (Crown Prosecution Service) has always had the right to take over and discontinue a private prosecution.
I used to work for BT post split from the Posties and when I commented that the procedure for IB/SD investigations was very rigorous i.e. all interviews taped two copies of the tape kept.
I said this is just like if the police where investigating your for murder and I was told ah well in the bad old days people used to fall down stairs on occasion
It's really not about the bugs. The bugs were unfortunate, but bugs happen. The problem is taking the word of the system at face value and not investigating further even when dozens of people's livelihood and freedom are hanging in the balance.
Oh, and also the bit about spending 2 decades covering everything up and trying to clamp down on the investigation rather than admitting you got it wrong, once again at the expense the subpostmasters..
"The problem is taking the word of the system at face value and not investigating further even when dozens of people's livelihood and freedom are hanging in the balance."
We do the same thing with breathalyzers in most of the US. No independent people allowed to inspect the system for bugs.
The breathalyser issue is immensely frustrating. In the few cases where the software has been allowed to be examined, they found egregious bugs. Not to mention that the one-size-fits-all measurement model is inaccurate for people outside a very narrow metabolic range.
The trouble is that if you speak up about it, people ask: "why are you defending drunk drivers?" It's like innocent until proven guilty flies out the window.
All sorts of police field tests have significant rates of false positives. People actually get arrested on the basis of such "evidence" all the time. The justice and law enforcement systems essentially operate on the notion that these things are "good enough".
I've just recently had a trooper make 4 "mistakes" in court and in official reports. One of which I believe was an outright lie. The system doesn't care. If this were any other witness they would be discredited, but because they are law enforcement, they get a free pass. The agencies won't even handle the complaints correctly or file this information as Guglio data for when future cases request it (if found to be unreliable they can lose thier job). The system actors (law enforcement and judicial) in most states have special privileges in keeping information private - so special in many cases that if a complaint against a judge turns up exculpatory evidence that you have no right to it. The reason they state is to uphold the integrity of, and the public trust in, the system. I would think transparency would do that better. The only way that transparency would hurt those objectives is when wrongdoing is ignored or the punishments are so lenient to offend the public sense of justice.
But hey, I'm just a stupid peon, so what do I know.
> The reason they state is to uphold the integrity of, and the public trust in, the system.
Here's another dangerous idea: the justice system has false positive rates.
A judgement is a test with two possible results: innocent or guilty. Every test has false positives and false negativies. For every judge, there is a chance they'll condemn the innocent or absolve the guilty. How likely is it? We'll never know.
The logic behind "innocent until proven guilty" is that one false positive is worse than ten false negatives. The principle is that freedom is such a basic right that removing it unjustly is a crime in itself.
Appeals and Innocence Project type data should give some idea. Some states even have official programs to investigate peoples innocence after conviction. So it definitely happens more than it should and there should be some data out there.
From my personal experience, it seemed like everyone I dealt with who was part if the system was either incompetent, made mistakes, or might have even been covering things up. The trooper I already mentioned. The corporal investigating him said the trooper's lie was just a misunderstanding, but offered no reasoning or evidence to support that. He also overlooked some other things that were violations of policy (they only cited him for one reg when he fit several). The Sargent who approved the citation said he couldn't do anything about it even if it's incorrect, which is wrong because he can take action based on what the LT told me later. The LT investigated a follow up complaint (the trooper never took the corrective actions listed in first complaint). He didn't follow the correct complaint procedures since he closed the complaint without any notes or findings. I had to call IA, then I assume IA called him since the LT called me a day or two later. He can't even answer basic questions like what laws he used to determine the rules of criminal procedure weren't broken and if he investigated the trooper's conduct as prosecutorial misconduct since he acted as the prosecution and knowingly held an incorrect charge and made several misstatements in court (that we have evidence for). The first magistrate we got gave a continuance. He was arrested on unrelated charges. The second magistrate gave another continuance and thought that we were calling him prejudice when asking to dismiss with prejudice (most magistrates in my state are not lawyers). The third magistrate was a retired police cheif who showed bias, wouldn't let us present a motion, misapplied the law, and was yelling at us to the point his face turned red and he was not understandable due to a lack of air. The ADA originally assigned to the case had an email complaint and the information to confirm that the trooper knew this was an incorrect charge and that the citation didn't even claim the elements of the offense were met, yet they took no action and allowed us to be subjected to those pretrial restrictions. At the appeal, the new ADA tried to misapply the law, with some success. They also told the court administrator/scheduling not to talk to us and not to accommodate any of our witnesses without their approval, which normal for other types of cases with standard discovery procedures, but those don't apply here. The judge actually contradicted his himself in some of his reasons for his determinations - at one point saying a trial de novo is a complete do-over so he won't allow any record from that trial to be used and later saying you would need a record of the previous trial in order to have the charge dismissed. He then misapplied the law and would not even look at, or accept as evidence, an official letter from a state agency that helps define an ambiguous term in the law, which would show us to be in compliance with the law. Even without the letter, the principle of lenity and the rules of statutory construction were blatantly ignored. There were also some paperwork and administrative mishaps, like some information not being recorded and later spending 30 minutes on a moot motion, or the court refusing to provide us with the amended citation with the new charge because "we don't have anything to give you" eventhough it's in the file. The other party was in violation of two similar laws the this one, yet the trooper and DA office decided not to investigate/prosecute even when she admitted being in violation for one offense. So much for the law applying equally.
So yeah, in my view the system has no integrity and is severely broken. I have little faith in justice actually being carried out. I guarantee many people have been convicted who were innocent. I think it's mostly due to the fact that innocent until proven guilty doesn't exist anymore.
"It's like innocent until proven guilty flies out the window."
Honestly, this is now the public perception (and the system) operates these days. I had a trooper recently hold a charge that he knew was incorrect and it carried with it pretrial restrictions that no other charge would. The state police say there's nothing wrong with subjecting people to pretrial restrictions under charges that they know to be incorrect. The attitude is "screw you, criminal" (just a summary offense).
Some states actually get it right and use blood tests. That means that some blood is saved if the defense wants to have it tested (evidence preservation).
"It's really not about the bugs. The bugs were unfortunate, but bugs happen."
Once upon a time, computer programming attempted to be a profession. Fortunately for all of us who write code for a living, we no longer have to live under the threat of that responsibility.
It's far worse than just having bugs - they knew there were bugs and covered it up even when they knew what impact it was having. That's the bit I find genuinely shocking.
BT = British Telecom. British telco, which used to be part of the government-owned Post Office, but was separated from it in 1981 and then privatised in 1984. The delivery services part of the Post Office (Royal Mail) was separately privatised in 2013; but the retail post office business (Post Office Ltd) remains under full government ownership, albeit most of the individual post offices are privately run by franchisees – and it was these franchisees who were being prosecuted
Scots law is supposed to require two separate items of evidence to bring a prosecution - doesn't seem to have protected them as there are a number of cases being reviewed in Scotland as well.
Some of these postmasters died before being able to clear their names. It is a huge miscarriage of justice and a national disgrace. I don't suppose any of the guilty parties will be punished though.
doesn’t each side call it’s own expert witnesses? and cross-examine the other sides expert witnesses? Doesn’t seem like it would be hard for the defense’s expert witness to show the possibility of bugs?
Fully agree with this, it is incomprehensible to me that these convictions required basically the say so of a government owned entity to convict people in private prosecutions (not done by the CPS)
While I agree more broadly, you cannot expect the Post Office to prove a negative. How would they prove conclusively that the software had no bugs at all under any circumstances? That's a pretty steep QA bill imho.
If their code is a mess and has errors, and there are people at post office that know this, they should be fined on the spot for false representation of contemp of the court.
If their system were up to date, written in a safe language, has unit tests and an independant review said it was solid, then it is just one acceptable piece of evidence.
What i dont get is - where was the money? Supposedly hundreds of people stoke huge amount of money, and none of them had it in a bank, bought a new car, or showed any signs of suddenly becoming wealthier. Where did the judge think the money go to, they ate it? How was this not suspicious?
The amazing thing about these cases are exactly how many shitty things had to happen (and did) for this to occur. Like you say, where is the money? And why didn't anyone spot that more sub-post-masters were getting charged than almost all other employee types? Why didn't anyone manage to reproduce the error? Why did managers hide the reports (who does that?)?
I personally think this is partly down to the fact people don't get state defence lawyers anymore in the uk. You could accuse me of fraud with zero evidence and I likely would have to plead guilty as I don't have 20k for the down payment for a lawyer...
What a shit storm. Now watch as nothing changes...
Agreed that you cannot expect the Post Office to prove a negative but, if they cannot prove a negative, they should not be able to use their computer system as evidence of theft and fraud. Especially if this is the only evidence they have.
They can actually use formal methods to prove that their software is bug free. This technique is often used in safety-critical systems to ensure that they function as-specified. As long as the specification is correct, the software system should perform to specification under all input conditions.
We should consider the cost of QA and of engineering process against the cost to these 39 people of their freedom and a large part of their lives due to an accounting error in the software.
Formal proof of program correctness tends to require that the software's purpose lies in a very narrow, and extremely well-defined problem space. The _Horizon_ software in-question is a general-purpose line-of-business system, which presumably has to react to ever-changing business requirements - that's probably the hardest space to implement formal-methods in - with little benefit for doing-so precisely because requirements change so often.
The places where you do see formal-methods would be in, for example, FADEC for aircraft engines, or an operating system process scheduler.
You don't need a formal proof for the entire system as a whole. You could simply have a formal proof of specific functions of the software. Those functions which will have their data audited, for example.
Correct - but this is a line-of-business application software: so I'm assuming that everything about it is subject to changes. The parts of the system that would be thoroughly spec'd out, QA'd (and maybe do have formal proofs) would be the underlying database system they're using which would be a third-party system like Oracle or Sybase or MS SQL (I just hope they're not using Progress/OpenEdge...).
The bugs are described as being part of the systems' payments processing system (as the Post Office does function as a financial institution, after-all) - but that at least some of the bugs were caused by ad-hoc work on the system (wot, no CI/CD to gatekeep releases?!) so I'm thinking this is just institutional incompetence - let alone a lack of software-engineering processes - so the idea of them implementing formal-methods for proving the system's correctness is laughable.
Formal methods require a comprehensive specification. Usually, if you have a specification comprehensive enough for formal verification, you already have 90% of the benefits, which is why it's (so far) only really useful in safety critical applications with a very small scope. I'm not going to take a huge risk in betting that the postal service didn't have anything resembling a serious spec in this case.
That wasn't the issue. Fujitsu, who operated the system, had access to post office branch systems with full access, but this was denied. The postmasters were prosecuted individually without good enough representation. The 2019 judgement is a good read about how the prosecutions happened and how the evidence was presented [1]. Really, formal methods and bugs were not issues, this was a system with humans in and someone decided that some of the humans should be blamed for issues, because the balance of power let them.
Sure, I can agree with that — it’s way too hard to prove a big system isn’t buggy. So then you also can’t use its output as evidence in court, right? You have to have other evidence that you can prove isn’t faulty. Can’t have it both ways.
This is the proof that how we write software is inherently wrong, if we allow innocent lives to be destroyed because we don't want to write it differently.
Imagine not doing inspections of new building construction because it would be costly.
That is a tempting conclusion, but consider if the software was 100% correct per some specification, and the spec was wrong?
No, the problem is greater than that. Decisions that affect people should not be made solely by computers or algorithms, and those decisions should be made transparent and auditable. If that leads to different/better ways of writing software, good. It's a larger societal issue though.
> This is the proof that how we write software is inherently wrong, if we allow innocent lives to be destroyed because we don't want to write it differently.
I think there's a lot of room for writing software better, including expanded source access for public systems and formal verification when critical.
But the failure in this case isn't technical, it's legal. It's rational to decide that occasional bugs in a mail software system are acceptable, and not worth the cost of designing a system's development around formal-verification. What's obviously insane is treating such a system as if it's bug-free beyond a reasonable doubt, and ruining innocent people's lives over it.
There are a lot of forms of gross incompetence and negligence that we're all fine with because they're so common. Failing to reason about software systems and their pitfalls, or consult with those who are capable of doing so, is an extremely-common and often-dangerous example (cf dumbass Senators grilling Zuckerberg with their 1970s understanding of how technology functions).
The blame here lies squarely on the prosecutors, judges, etc who are responsible for these verdicts. They should be ashamed of themselves.
Unfortunately, that sounds straight-forward but isn't:
Q) Did you or any of the people you got to examine the software found any way that what the defendents said was true?
A) No
Q) Then you are guilty beyond reasonable doubt.
I think the bigger issue here is around the power that a large organisation wields to duck and dive and use corporate tricks to manipulate how it played out. For example, the fact that so many people had been accused could have been analyzed if it was known e.g. Last year 5 convictions, this year, 700!
If you're going to convict someone of stealing £59,000, the very first thing you should have to show is that £59,000 actually got stolen. If there is reasonable doubt that the crime took place, no one can be guilty beyond that reasonable doubt. If the defendant claims the computer system got it wrong, it's not enough to say you are unaware of bugs, the prosecution should have to show that the computer's output was consistent with the results of doing the calculation by another method.
Not only that £59,000 was actually stolen but that the accused received the money. There was no evidence that the post masters charged ever had the money in their possession!
> Not only that £59,000 was actually stolen but that the accused received the money.
Can only speak for English and Welsh law, but this isn't accurate. Theft is prosecuted under the Theft Act 1968 and does not require the accused to actually receive the goods or money stolen. All the accused need do to "appropriate" property is assume the rights of the owner e.g, if the accused had access to someone's bank account and they sent money to a third-party, that's still theft because they assumed the rights of the owner (to transfer the money) even though the money didn't go to the accused themselves.
What you are both arguing doesn't match up with the facts. It was proved beyond reasonable doubt multiple times. The "proof", (which we now know was flawed) was that the system had shown that what they had sold didn't tally with what was sent to the Post Office, to refute that proof, the defence have to show another plausible explanation.
Yes, if they could have proved they also received what was stolen, that would have been a slam dunk but there are enough plausible reasons why they can't find the money. Maybe it was given to friends and family as cash, maybe it was used to gamble or to pay off some criminal.
It isn't much different than somebody saying, "you did it because we found your DNA". The Courts or Jury are inclined to believe it because "science" and if the defence are not on their game enough to show how "because DNA" is not always watertight, the defendent is seen as guilty beyond reasonbale doubt.
> The "proof", (which we now know was flawed) was that the system had shown that what they had sold didn't tally with what was sent to the Post Office
No, it claimed that what they had sold didn't tally, a claim they never proved. The defense put forward another plausible explanation - that the software was incorrect, and the prosecution obviously didn't prove the software was accurate.
Even if the computer was right and there was a genuine discrepancy in the tally, you then need to prove that this person was the one responsible for it. Certainly in this case, there couldn't possibly have been sufficient evidence to prove they were the ones that did it if it was never done to begin with. Absence of evidence isn't evidence of absence, but it sure as hell isn't proof of presence.
In the "we found your DNA" analogy, you're finding my DNA in my workplace where nothing has actually gone missing - how does that prove I am guilty of theft?
> Yes, if they could have proved they also received what was stolen, that would have been a slam dunk but there are enough plausible reasons why they can't find the money. Maybe it was given to friends and family as cash, maybe it was used to gamble or to pay off some criminal.
If this was one or two cases, then sure, maybe they were really smart about hiding the money. However, there were hundreds of convictions. What is the more likely explanation?
"The "proof", ..what they had sold didn't tally with what was sent to the Post Office"
Post office looses packages all the time, should someone go to jail for that too?
If their stuff doesn't tally, they are disorganised, they loose stuff or have idiots. Thats their problem. Maybe it's post office employees stealing shit.
Why do we immediately assume postmasters have abything to do with it without a shred of evidence?
> but there are enough plausible reasons why they can't find the money
There is a strange presumption in here. It is true that lack of evidence doesn't always means there's evidence that there was no crime. But that shouldn't matter. A crime should only be prosecutable if it is demonstrable. We shouldn't say "oh, well the prosecution had a really hard case, we should just convict this person anyway because it wasn't fair to those lawyers." That's such a perverse way of reasoning about it.
Glad to see them finally have their names cleared, and can only hope prosecutions will follow as a result, utterly shameful how the Post Office, Fujitsu and others behaved. For example:
> A Fujitsu programmer from the time, Richard Roll, who would become a key witness in the sub-postmasters’ high court case against the Post Office in 2019, told the Eye that Horizon was one the company’s few profitable contracts. Among other private sector deals, it was also lining up a key role in the mother of all government IT splurges, New Labour’s £12bn NHS IT project (Eyes passim ad nauseam). Fujitsu could ill-afford either bad publicity or the penalties that came with software faults. “We would have been fined,” said Roll, who worked at the company between 2001 and 2004. “So the incentive was to pretend it [software error] didn’t happen”, while running “a constant rolling programme of patches to fix the bugs”. Fujitsu “would basically tell the Post Office what they wanted to hear”. So prolific did Roll’s bug-fixing team become it won the company’s President’s Award for outstanding corporate contribution in 2002. And the quick-fix, ask-no-questions approach that suited Fujitsu financially enabled the Post Office to hold the line that blame for all branch shortfalls must lie with the sub-postmaster.The Fujitsu insider concluded that errors leaving sub-postmasters out of pocket were inevitable. Could that mean hundreds of them? “Given there were [about] 20,000 post offices when I was at Fujitsu and the sort of problems we were dealing with all the time, yeah,” he told the Eye. “Sounds reasonable.”
>For the first 10 years of Horizon’s existence, transaction and account data was stored on terminals in each branch before being uploaded to a central database via ISDN. Our source says this part of the system simply did not work.
>“The cash account was a piece of software that sat on the counter NT box, asleep all day,” he said. “At the end of the day, or a particular point in the day, it came to life, and it ran through the message store from the point it last finished. It started at a watermark from yesterday and combed through every transaction in the message store, up until the next watermark.
>“A lot of the messages in there were nonsense, because there was no data dictionary, there was no API that enforced message integrity. The contents of the message were freehand, you could write whatever you wanted in the code, and everybody did it differently. And then, when you came back three weeks later, you could write it differently again.”
And down further
>Speaking to Computer Weekly in 2015, the anonymous source told us: “The asynchronous system did not communicate in real time, but does so using a series of messages that are stored and forwarded, when the network connection is available. This means that messages to and from the centre may trip over each other. It is perfectly possible that, if not treated properly, messages from the centre may overwrite data held locally.”
>Four years later, former Fujitsu engineer Richard Roll wrote in a witness statement to the High Court: “The issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at branch level.”
BUT the most important part
>So far, nobody at the Post Office or Fujitsu has been held accountable
The most important part is that the PO used these actions to claw back "stolen" money from its postmasters. This money appears to have ended up in its profit and loss account.
If true this means that instead of the postmasters stealing from the PO, the PO was stealing from its postmasters.
There's been at least one claim - in the Daily Telegraph, so questionably credible, but never mind - that a document exists proving that senior management were aware that the accusations against postmasters were untrue, but carried on regardless.
If that document exists it changes the narrative from accidental tech failure and management incomprehension to something less wholesome.
BBC (broadcast so don't have link) said that under the previous CEO that an investigation was shelved into the accusations. Given the number of accusations I have to wonder if there was a cover-up.
Jail sentences, bankruptcy and suicide has been caused, management that oversaw this need to face prosecution.
Written from the POV of a former BT billing systems developer - The system was designed (fucking badly) before the widespread existence of ADSL.
This is what happens when you outsource core financial systems to low cost bidders with dubious tech chops building a message queue system is not fraking rocket science at this point.
Back when I worked on the ground up billing system for Telecom Gold (aka Dialcom) we did this as the existing mish mash of dodgy code that Dialcom offered (Sorry Eric) was not up to standard.
We had large amounts of internal auditing built in and we tracked discrepancies to the Penny.
I think a lot of "normal" people like the idea of holding corporates accountable but how would that actually work?
The CEO blames one of their directors; the Director blames the supplier; the supplier blames the requirements documentation; the Business Analysts blame the culture for creating confusing and conflicting requirements.
Yes, you can hold the organisation accountable but then the people who worked there back then are long gone, they don't care if the Post Office gets fined £500M.
You only have to look at the enquiry into the flammable cladding scandal which was entirely down to fraud, yet, there are people who have not been arrested over their misrepresentation of their products.
> I think a lot of "normal" people like the idea of holding corporates accountable but how would that actually work?
Exactly which specific problem is "holding corporates accountable" trying to fix?
If it's that postmasters were being falsely convicted, then the way to fix that is to raise the burden of proof significantly. I hope this case has done that, and next time a court will not accept "computer says so".
With that fixed, the corporates would have to take the (falsely reported) losses; they wouldn't be able to pass it on to the postmasters like they did. Then the consequences of the problem will remain with the people responsible.
The problem is that it is possible to design malicious systems which through incentives, ensure that illegal acts will take place, yet only low-level actors are ever punished. The people who architected the systems and made the decisions statistically guaranteeing illegal activity escape punishment through plausible deniability and abscond with their ill-gotten gains.
Besides this scandal, see the failure to punish any executives after the 2007 crash, or Carrie Tolstedt and John Stumpf of Wells Fargo who even after clawbacks retired tens of millions of dollars ahead, etc.
Our animal brain doesn't perceive this situation as what it was due to the layers of obfuscation: a gang of vicious criminals kidnapped innocent people for months or years at a time, repeatedly, for decades, covered it up, robbed those people, ruined them financially, separated parents from children. Oh but they used perjury and fraud to get the state to do it for them. The mafia wouldn't even have the balls to do something like this. That is what actually happened, though of course, obscured behind layers of legalese and obfuscated responsibility it is difficult to get the emotional response it deserves to trigger.
The individuals need to be held accountable and do jail time. "Just doing my job" is literally not an excuse for breaking the law!
Everyone who knowingly covered this up at least committed fraud and/or perjury, or were accessories to fraud or encouraged or even financially remunerated people to perjure themselves(lower level people getting promoted for lying in court).
Their perjury led the government(in the name of us, UK citizens) into immorally, if not illegally, depriving other citizens of their liberty and money. If the legal framework to punish this as fraud does not exist, create it. If someone knowingly and with premeditation lies to the government and gets other people in trouble, they should get back that trouble sufficiently ramped up to act as deterrent. 10x 100x the fine, fine as percentage of lifetime net-worth whatever. If people knowingly and REPEATEDLY get people thrown into jail, all those people should be doing the amount of jail time they fraudulently inflicted on innocents. If the problem is the organized plausible deniability nature of the crime, I'm sure there's some racketeering or organized crime laws that are applicable. This whole thing literally became an organized crime organisation.
The CEO(s) at the time are the chief persons responsible.
If there are no consequences for harm, more harm will be caused. By not punishing wrong-doers and shortcut takers you ensure that more wrong-doers and shortcut takers will rise to positions of power, because they are able to outcompete ethical players.
White collar crime is literally destroying modern western society, if not in fact(though I believe in fact as well) then at least by ruining popular perception of our (necessary for a society to function) elites. We need to clean house.
The quick and dirty way is to somehow tie their power/privileges/financial-situation to that of those who they have power over. (And make it stick for many years.)
There's a big missing culture of fixing problems in corporations. Which of course must start with acknowledging the problem. Which of course means that people reporting problems shouldn't face negative consequences. Which means that the current cultural gap is not just a nice empty void, it's an actively hostile roiling psychological chasm of corporate warfare.
So if random CEO knew about some problems that actively harmed the employees and did nothing, and later a court says that the company did wrong, the CEO automatically has to pay some fines too.
And it should be possible to share (but not completely delegate) this responsibility down the corporate hierarchy, to incentivize executives/VPs/managers/team-leads to do the right thing.
Of course this would need a political culture that is motivated to develop, fine-tune and enforce such a framework. ¯\_(ツ)_/¯
one thing that ive noticed/realized is that perhaps associating hierarchy with division of labor/responsibility is perhaps an anti-pattern...
what if the ceo was responsible, but was working under the managers "for them" what if managers worked at the same level as thier team whos role is to make the team productive and support them (or even the team could fire thier manager) etc etc
i wonder if disassociating those two (responsiiblity/hierarchy) might be a step towards fixing these kinds of issues...
The concept of "Servant Leadership" is not exactly new [1], Many people are not in the position to argue with their boss when they frame it as an adversarial relationship - That you owe something to the boss for your pay, that it's somehow a privilege they are granting you to work for them. There's no way to prevent those kinds of leaders from having a successful organization except by refusing to work for them and out-competing them, and fundamentally: Crime does pay, sometimes for long enough to starve legitimate competition out.
Hm. My experience is that most of the time as responsibility is diffused in big organizations people are motivated to just stay quiet, weather the storm, don't rock the boat. And the typical "matrix management"-like organization structure (where everyone has many different managers, eg. there's a project manager, there's a technical manager (let's say engineering manager), there's a [human] resource manager, there's a team lead [a non-commissioned manager]) exacerbates this.
> even the team could fire their manager
This would likely help a lot with the Peter Pans who ended up promoted to managers but are terrible at managing.
At some (or perhaps more than one) point there was someone who was responsible for ensuring that the system put in place complied with requirements and that it was functioning as intended. They didn't do their job. They can point their finger any which way, but that won't absolve them of dereliction of duty.
That would QA department. And QA department is not exactly the most respected department in technology. I have seen numerous times where issues raised by QA are ignored, or even worse they get code to test 1 day before the release.
The issue lies with leadership. Start with CEO for creating a culture where safety and quality is ignored. Go down the chain only if there is considerable proof that someone under them ignored corporate orders and delivered buggy software.
I don't think the idea is to hold the "corporate" accountable. If it's true that the senior management at that time decided to cover up the failure and ruined the lives of hundreds of people, and knew full well that they were doing, they should be punished, even though they might no longer be working in the company. Simple as that.
Blame lies with CEO. Unless CEO can show there was insubordination that is CEO wanted to do the right thing but director ignored CEO and ordered their reports to ignore CEOs orders.
One interesting idea is to have a corporate "death penalty", although that wouldn't necessarily work here as well because it's not a private corporation.
I don't think the state of broadband can be the cause here. Banks, supermarkets and even GP surgeries were able to support complex accounting systems or patient records for decades.
Seems like the Futisu team running Horizon decided to reinvent everything badly.
The first bizarre part to me about this fiasco is that accounting, as a discipline, is one that is designed to catch errors. Put it another way: it assumes errors will occur. This is why in shops, for example, you'll have manual stocktaking (ie let's verify what's in the store is what the computer thinks is in the store) and in any business you'll have reconciliation processes to find and remedy errors.
This highlights a key part of systems design. A key question you should be asking is: what happens when this fails? Note that's "when" not "if".
So something like Horizon should be used to flag cases for reviews. If a branch is found ot have a cash shortfall suggesting possible theft then there has to be a reconciliation possible to identify if the computer system was wrong.
Bugs happen too. How do they ever have confidence in the system and fix bugs if they can't determine if a given flag is a false or true positive?
But instead the system's output was taken as gospel with no possibility of verification. I'm of the belief that if you can't verify anything the system outputs, particularly for something in a discipline so used to verification as a concept, then that signal is worthless. The fact that convictions happened as a result of this is a crime. This is the UK and not the US so sadly that compensation will probably be limited to nonexistent.
As an aside, this is exactly why electronic voting should be outlawed. You need paper ballots (that can be counted electronically) as a verification measure. And the fact that we even have to debate that makes me sad.
Nationally regulated, sure. Verified with a physical copy (or a different system), sure. But banned altogether? You might as well ban everything in the world that is digital, as none of them are fool-proof.
Voting isn't even that important. The wrong guy gets picked, what happens? Same bullshit as if the right guy got picked. If your choices are "Hitler" or "Jesus", then your system is just fucked up, and making voting fool-proof isn't the way to fix it.
In addition, electronic voting would be a boon to democracy. It would provide another avenue for maligned minorities in remote areas be able to vote, when things like paper ballot voting in the middle of a pandemic might fail or be error-prone (esp. when a fascist fucks with the postal system), or local authorities enforce racist requirements like a physical ID card.
If you vote on a touch screen and it prints out a paper ballot, that's fine as long as that's a legible ballot, like not just a QR code or something. The voter should have confidence in the output.
Likewise, if you use a pen or pencil to fill out a ballot that then is counted electronically, that too is fine.
In both cases there's a paper ballot as a source of truth and that's what's key.
> Voting isn't even that important. The wrong guy gets picked, what happens? Same bullshit as if the right guy got picked.
If voting is unimportant, why do you care about racist requirements for physical ID cards? Perhaps there might be some sort of connexion between the two!
It's more important that you are able to participate than what the result is. Better to have an insecure system where 10 million people get to vote, than a secure system where only 10 people get to vote.
Software is in the walls. At some point legislators are going to come and ask the question how we stop things like this happening and if the Fujitsu's of the world don't have an answer then we can expect regulation that will likely embed practices that don't help.
I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things. This culture ruined these peoples lives. Things must change. This isn't a unique issue to Fujitsu it is something most of the software industry is doing, this story could be about just about any piece of software.
> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things. This culture ruined these peoples lives. Things must change. This isn't a unique issue to Fujitsu it is something most of the software industry is doing, this story could be about just about any piece of software.
Damn straight. I'm really big on software Quality. It's kind of my driving passion.
It has been my experience, that an attitude of Quality is actively discouraged in today's "rush to a crappy, lashed-together-with-baling-wire-and-bandaids MVP" SV culture.
We glorify and make heroes of those that deliberately publish garbage, but make money doing so.
When we look to an industry to police itself; it never does. But the rules and regulations applied from non-domain-expert politicians are often ineffective, burdensome, and really only apply to a bygone era (See ISO 9001/CMMI).
Both. It's very well geared towards maintaining a certain standard of quality and predictable project throughput in rather well defined projects, and it certainly makes the job easier in procurement, but it's totally detached from what the SWE world is outside of that.
I can see the point of such models in certain areas, like military, aerospace, naval, or, to stay on topic, Horizon, where dev is outsourced, somewhat critical, specs rather set in stone, and non experts need to measure how capable an organization is to deliver, but for anything else it just feels like unnecessary meta-management that brings significant organisational and development overhead.
They had a good idea, but they applied "old world" thinking to it.
The single biggest issue with software development, is that it is incredibly dynamic.
Static solutions don't work, and CMMI is a very static solution. Sadly, a lot of quality practices are static.
Dynamic solutions are really difficult to get right, and tend to depend on a lot of hard-to-quantify variables, like the experience and talents of individuals on a team.
For example, I am quite good at designing fairly complex systems, as long as I am doing it alone. I can hold some fairly ambitious designs in my head; which allows me a great deal of flexibility. I can start with a fairly "fuzzy" architectural model (I call it my "napkin sketch"), and begin a project fairly quickly. As the project progresses, I can apply some massive structural changes, and pivot fairly easily.
However, the minute I need to communicate this plan, the whole shooting match comes to a screeching halt.
Team overhead is a really big deal, and I believe it is seldom factored into our plans, in any kind of realistic manner.
We have some marvelous CI/D tools at hand, but the execs are the ones that push to release before ripe, and they won't let things like auto-test failures get in the way of MVP.
There was a comment here, some time ago, that was made by someone that proclaimed themselves to have started and successfully exited a number of companies. It went something like "If you do not get physically sick, looking at the code in your MVP, you are spending too much time, worried about code quality."
I think that's a pretty good summary of today's startup zeitgeist.
That sounds like bad engineers. Set your CI/CD to fail to merge code if any automated tests fails. We're not talking weeks of extra work to get tests to pass.
I think that as soon as we say "zeitgeist" we're abandoning all attempts at understanding the full picture. There are huge numbers of tests being written all over the place, particularly with modern software tooling. It's the older stuff (e.g. Oracle Forms) or low code stuff (e.g. Bubble, Dynamics) that are hard to test where the biggest issues crop up, IME, as those technologies will be picked by people who don't value testing.
The main issue is management; not engineering. I'm actually quite chuffed at some of the amazing tools I have at my disposal, these days (like profilers and code coverage tools, as well as runtime debuggers). I "grew up" in tech from the early 1980s, and things have changed.
The one thing I miss are ICEs (In-Circuit Emulators). Those were badass, but processors are so massive, and so fast, that a real ICE would probably cost a couple of million dollars, and be out of date by the time it hit the market.
Tech can't fix bad managers, and money is like Miracle-Gro™ for bad management, if it incentivizes rotten quality.
> The most important bug is that the software doesn't solve the problem that you have.
And we should add:
Unless we can't do so without introducing any additional problems, while solving that problem in a manner that truly solves it; as opposed to making it appear solved.
We really are often best off, with the problem, if the cure is worse than the disease.
When I was younger, we had a saying:
To err is human, but it takes a computer to really fuck things up.
This seems more an issue of bureaucratic incentives than software. Fujitsu wanted to hide bugs to look better for future contracts. The Post Office wanted to hide bugs to deflect blame from central leadership and be able to scapegoat people at will. The judicial systems seems to have either not cared or had incentives for some quick prosecutions.
Software doesn't exist in a vacuum and software will never be perfect. Trying to solve systematic problems by holding one part to impossible standards will just make things worse rather than better.
I think we're already over the edge of that, it's more urgent than you think.
A couple of insensitive Facebook posts gets you dropped from consideration for a job... no matter how long ago and how much you may have matured in the meantime.
Google implements FLoC and cohorts start identifying political leanings, medical conditions, mental health issues, anything that's legally potentially discrimination territory... how do you know that someone deduced a cohort topic and denied you <something> based on that...
Tip of the iceberg. Data aggregators already have opaque records on probably everybody alive, just find the one with data about your person of interest.
This needs to be a complete change of awareness and ethics and global law... otherwise we're going to have the movie "The Circle" come completely true as opposed to being just around the corner.
> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things.
This drives me up the walls. At my last job (food ordering startup the CEO had the attitude that releasing code that was 95% functional was Okay, remaining issues could be fixed as we went along.
As a result, one developer overlooked a bug that cost the company €300,000, loyalty discounts weren't being deducted from payments to take-aways. They then had the cheek to demand take-aways pay them back.
Then they launched a major upgrade to the system at 5pm on a Friday - two hours before their busiest time of the week. It collapsed a few hours later and it was impossible to roll back because they didn't include a roll-back SQL script for the DB. It took till the following Tuesday to fix it.
The DB schema was all over the place and as a result it was slow. Entity Framework couldn't handle it and the SQL it was generating was terrible. Me being the only one with decent SQL knowledge had to replace all the bad EF queries with raw inline SQL.
Despite this, they still carried on deploying without a care in the world. I was told to stop moaning about QA. We didn't have QA or testing staff, the CEOs attitude being why pay for QA staff when our clients will do it for free?
I worked at a place that internal customers complained QA took too long, so IT said fine, we won't do any. Then they complained the software didn't work... people sure can be stupid.
An oath isn’t going to do anything without any way to enforce it legally. The Hippocratic oath is neat but the real teeth are in enforcement against malpractice like civil and criminal lawsuits and a licensing body. You see similar things for lawyers and certain engineers (in commonwealth countries “professional engineer” is a restricted title like Lawyer or MD). Note that just doing that won’t solve all problems either. These licensing bodies regularly publish enforcement actions, so malfeasance continues. Nominally they can help whistleblowers but, as with all regulatory bodies, there’s always a risk of regulatory capture making such actions still peril-filled.
Moreover it’s not even clear this particular work even fall under traditional definitions that would required a licensed engineer as those deal with public safety (bridge construction, buildings, etc) and something like this doesn’t really. We’d need an updated definition that takes into account the software needs of the world (privacy and security, etc).
"software developers… need to have an "oath", like the hippocratic oath."
More importantly, the employers of software engineers must have ZERO option to emply a software engineer (anywhere on earth) that doesn't have the same oath.
Doctors have a monospony on their services that makes their oath work: the hospital manager cannot just go hire un-oathed doctors.
The practice of medicine was not a licensed endeavor at its inception, and that changed over time. With that in mind, what makes you say "Never going to happen in software. Ever."?
Doctors don't follow the hippocratic oath in practice, it just isn't a real consideration. If they did none of these covid long haulers or ME patients would have been tortured into worse conditions, nor would all those mentally ill patients have been locked up. Medicine treats the oath like software developers treat most best practices that reduce bug counts, as a nice to have but no one has time for.
> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things.
The phrase "move fast and break things" should be seen as cautionary, not aspirational.
How do you know this culture existed 20 years ago when this system was developed? It is almost certain that a corporate developer in the 1990s would be 100% waterfall.
The issue is very often related to massively complex corporate requirements (the Post Office makes me cringe, even today, with the complexity of their postal system) and then coupled with the ever-present need to keep costs low, especially when designing something so complex.
I doubt anyone building this thought it would be OK to break things!
You are correct. I've seen the Horizon design documents - they are very detailed, thorough and complete. The system was not designed in an Agile manner of iterative design / build first - document later.
Throwing innocent people on jail based on lies (with bonus, corrupt government officials colluding with foreign entities) is the problem here, not software bugs.
Software bugs happen. The trick is to have proper management of the release that takes into account the inevitability of bugs while incentivizing bugs to be identified and fixed without the stakeholders of the project being in a position to have to defend a project as if it is perfect.
I cannot imagine how it must have felt being under the boot of an entire government and it’s corporate partners due to a bug. This is why we are important. A poorly managed IT system with bad incentives puts lives in danger. It is a literal threat to the safety of society. This cannot be stressed enough.
> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things. This culture ruined these peoples lives.
I think people see a false dichotomy between making things quickly and making them safe. The fact is in the development of any complex thing, you're going to have bugs, and generally that's okay. But things should be designed to fail safe. Making something that throws errors when something unexpected happens is actually faster and easier than trying (and possibly failing) to handle edge cases; had Fujistu taken that simpler, easier approach then all this pain and suffering would have been avoided.
I still think https://xkcd.com/2030/ has to be taken seriously. You can put a whole load of verification effort into your software, which will undoubtedly make it more reliable. But you are still likely to have some kind of corner case where it breaks down. Software is complex enough for this to be universally true.
The key is how we respond when the software fails. The https://en.wikipedia.org/wiki/Therac-25 case shows an example of what not to do - when hospitals started reporting their machines giving lethal radiation doses to people, the manufacturer doubled down on the computers-are-infallible rhetoric, where they should have put every last effort into investigating. Likewise, the post office should have noticed that a rather excessive number of postmasters were apparently fiddling the books, and investigated. Instead, after it was fairly obvious that the computer was wrong, they pushed the computers-are-infallible line right through the courts, and that is what earned them the "affront to justice" judgment.
The key is how we respond when the software fails.
I agree, but if the first step to solving a problem is understanding that it exists then the first principle here must be to acknowledge that software systems are fallible and therefore any surprising or reasonably contested result they produce should be treated with proper caution until further information can be gathered.
So many of the problems we see when modern technology goes wrong start with assuming it didn't. At that point, it's not even about how you respond to the failure, because you're denying that the failure ever happened. Big software companies with considerable lobbying power seem to be particularly good at convincing people who aren't technical experts, including most politicians, judges, juries and reporters, that this is the case.
A corollary to this is that we desperately need more technological awareness among our politicians, lawyers, journalists and other relevant professions. Tech has become too big to be a minor issue you delegate to some random advisor in a basement office. It affects almost everything we do today, sometimes profoundly, and failing to understand that will inevitably lead to some horrible outcomes as we've seen all too vividly today.
I agree with the sentiment but the example taken for software in this xkcd is wrong. There is a fundamental problem of trust when using software for voting systems that is not linked to the reliability of software but to the nature of voting systems and the properties we want.
Well, there are two problems and which is the fundamental one depends on your prior assumptions.
Some would say it's impossible to build a secure electronic voting system, even if your supplier and their employees were completely trustworthy because between physical tampering, state-level adversaries, the state of the art in software development and the impossibility of proving a negative, such security has never been seen before.
In other words, that it's an unsolvable technical problem.
Others would say it's impossible to build a secure electronic voting system even if we were capable of creating flawless bug-free and tamper-proof software and hardware because the supplier will always be able to introduce undetectable bugs if they want to, and no supplier can ever be perfectly trustworthy.
In other words, that it's an unsolvable social problem.
Yeah, electronic voting is essentially like having a person in the voting booth that you have to tell your vote and trust that they will tally it correctly. [1]
It doesn't matter whether voting machines are actually secure, they probably mostly are right now, but whether a layperson can have faith in the system.
Paper voting is very secure if you involve people from opposing parties in the process and attacks are not very scalable. Most people can think of and understand mitigations for certain kinds of attacks. And if paper voting is too expensive for your country, you have bigger issues. [2]
[2] That said, I don't see how secure electronic voting can possibly be cheaper than paper voting. For voting machines to be secure, you have to manufacture them in a very audited manner, with little to no foreign sourcing of parts, you can't leave the machines unattended for long periods of time (aka, reusing them between elections is probably a no-go) and you have to build them in manner that is secure against voters tampering with them in their private booth.
> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things. This culture ruined these peoples lives. Things must change. This isn't a unique issue to Fujitsu it is something most of the software industry is doing, this story could be about just about any piece of software.
This won't change until executives go to jail.
A few years ago, we were fighting against tight deadline and skipping unit tests, QA, processes, etc. Someone brought up one of the recent major breach (Equifax?). Developers started to say that people will go to jail. Basically, devs were using this breach to imply that they will personal responsibility for releasing a product that might have security flaws. Our director laughed and said no one will go to jail and if our product ever got in trouble, they will personally take responsibility.
> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things.
This is extremely domain dependent, and should be handled as such. And in some cases it already is - look at the testing / verification space shuttle code goes through vs your friends cat video side project website.
> I don't think we take software reliability seriously enough, most of our focus is on speed of release, ever quicker cycles and it being OK to break things. This culture ruined these peoples lives. Things must change.
I disagree. The needed quality of your software changes dramatically depending on what it is used for. Something that helps someome semi-automate their workflow where bugs just mean they have to do things manually (as things that are wrong fail quickly and obviously) is something where what matters is speed of release and features. Something calculating how long people should be in prison needs to care about quality.
Anyway, the courts are the ones that have the biggest share of the blame here. Believing an unreliable witness for the prosecution is a common cause of injustice.
> At some point legislators are going to come and ask the question how we stop things like this happening
If the problem is the software, then use less software. Perhaps we shouldn't simply take it as a given that moving processes into software isn't always the right move?
This is unreal. Shitty software sending people to prison without anyone in the process considering what exactly is the likelihood of hundreds of postmasters simultaneously becoming thieves overnight.
Yeah, this is the part I'm having trouble understanding. A few people, sure. But all these postal workers committing fraud, with many insisting there must be something wrong with the software? How did this not get discovered before they were all convicted and sentenced?
And according to the article, the full number may actually be something like 900 people.
>Campaigners believe that as many as 900 operators, often known as subpostmasters, may have been prosecuted and convicted between 2000 and 2014.
How do you make this mistake almost 1000 times over 14 years before someone suspects the system data may not be quite right? Also, even if you do completely believe the data, how can you convict them all without additional supporting evidence, like new purchases that don't seem to fit their salary, suspicious bank transactions or balances, records of unusual system access or them actually manipulating data, etc.
It pains a very bad picture of the Post Office, including:
- an expert witness from Fujitsu, who developed the system, "had been aware of at least two bugs which had affected Horizon Online[...], but had failed to say anything about them or about any Horizon issues in his statements";
- POL arranged a number of conference calls to discuss problems with the system; "instruction was then given that those emails and minutes should be, and have been, destroyed";
- "there was a culture, amongst at least some in positions of responsibility within POL, of seeking to avoid legal obligations when fulfilment of those obligations would be inconvenient and/or costly"
Further, once a number of convictions had been secured, the Post Office then used those convictions in later trials as evidence that the Horizon system was robust and reliable.
All in all, a prima facie criminal conspiracy by the Post Office.
There must be two source of independent evidence for someone to be convicted of a crime. I'll be interested to see (if there's genuinely no corroborating evidence beyond the computer records) how many prosecutions went ahead north of the border.
Given this appeal took place in England (and not in the Supreme Court), it was all English verdicts which were overturned as I understand.
The requirement for corroboration in such a situation would probably be met by having someone "speak to" the digital evidence and audit trail.
For example, if you have CCTV evidence, the CCTV is one piece of evidence, and it would be corroborated by a witness statement of the victim identifying them from the CCTV.
Corroboration is an important and useful safeguard, but I don't think it would necessarily have outright prevented this. Perhaps it would - maybe it would have raised the bar on scrutiny of the evidence, by there being a general higher expectation?
Hmn possibly. I suppose I am interested to see if there is a practical difference because there's some debate about whether corroboration is a good thing to have or not, when you can have one piece of evidence (like DNA evidence) which is very high certainty.
I'd expect there was prosecutions north of the border seeing as the post office is UK-wide so be good to see how they went.
>>considering what exactly is the likelihood of hundreds of postmasters simultaneously becoming thieves overnight
I mean, I don't think anyone assumed they suddenly and inexplicably became thieves, just that the fancy new software finally caught people who have been scamming the post office for years. Obviously the software was completely wrong and it's criminal what happened to those people.
yes, this reasoning does make sense. but given the human cost it should only make sense if there's a significant prior: in most of these cases there was no previous evidence whatsoever, just a new system, and boom, thieves.
I think the core point here is how imbalanced this process was: postal system builds a new accounting program that shows money is missing. these people were convicted solely on the evidence that software said so, there was no burden on them to show that the money was actually missing. I mean, hard for me to grasp how is that possible. anyone can write a program that shows something. how is this sufficient proof to send people to prison? does it not need to touch some objective reality at some point?
Yeah I mean if your brand new software discovered that a retail shop was suddenly missing £50k/month in income, surely you'd do full inventory to confirm £50k worth of goods is actually missing. No idea how you would do that in a post office, but I guess take an inventory of stamps and any other services sold?
This would normally be the role of a forensic accountant.
My suspicion is that the Post Office wanted to do this "at scale" and "automate", and just assumed blindly their own records were accurate, because well... They must be!
Had they actually tried to investigate these as one by one offences, you'd gather evidence of individuals concerned making huge cash transactions to buy expensive cars and holidays. And when you didn't find any evidence of this unexplained enrichment (as there wasn't any), your investigator would point this out, and you'd realise you didn't have a case.
Similarly a photograph of the subpostmaster getting into their outright-owned Lamborghini would have been useful evidence there. The absence of any of the evidence of this enrichment seems absent throughout. Let alone the detailed forensic accounting to determine what was actually taken. I suspect the issue was they simply didn't have any way to tell what should have been there, other than what the defective horizon system said... They were trying to run at national scale, without enough ground truth information to validate their assumptions and detect the issue.
I agree. My first thought on hearing this was that they'd look at the priors and realize there had to be a mistake.
My second thought was that most accounting departments I've worked with actually wouldn't do that, would blame fraud, and then would congratulate themselves at how much better they've gotten at detecting it!
This reminds me of what happened after 9/11, the fear of dirty bomb was all the rage so the US government deployed a network of Geiger counters. They arrested a number of dangerous dirty bombers, all of whom were cancer patients spotted by the detector at the subway station nearest Johns Hopkins radiation treatment facility.
At least when my wife hit that in the Shanghai/Pudong airport (residue from a heart scan, not cancer) they resolved it in a few minutes of talking.
On the other hand, I think Shanghai didn't check well enough--there was one simple test they could have done but didn't: Hand held geiger counter, see what's hot. Body equally hot, baggage not hot, it's medical.
They implemented the system without even thinking of the false positives. Eventually they added that to the procedures, but they harassed quite a few people before that happened. Cancer patients on top of that, many of whom were probably half dead already.
It seems to me that's an awful lot of stupid on their part.
Patient claims medical--call the facility and ask if they should be hot.
However, having read about their stupidity I would be inclined to get a card from the facility even if I didn't expect to be going anywhere with radiation scanners. (My wife had a card--which was sitting at home in the pocket of the jacket she had planned to wear. She changed her mind on flight day and didn't think about the card until she tripped the scanner. Note that she probably had an easier time of it than a typical tourist would have as she speaks Mandarin at native level.)
>For the first 10 years of Horizon’s existence, transaction and account data was stored on terminals in each branch before being uploaded to a central database via ISDN. Our source says this part of the system simply did not work.
>“The cash account was a piece of software that sat on the counter NT box, asleep all day,” he said. “At the end of the day, or a particular point in the day, it came to life, and it ran through the message store from the point it last finished. It started at a watermark from yesterday and combed through every transaction in the message store, up until the next watermark.
>“A lot of the messages in there were nonsense, because there was no data dictionary, there was no API that enforced message integrity. The contents of the message were freehand, you could write whatever you wanted in the code, and everybody did it differently. And then, when you came back three weeks later, you could write it differently again.”
And down further
>Speaking to Computer Weekly in 2015, the anonymous source told us: “The asynchronous system did not communicate in real time, but does so using a series of messages that are stored and forwarded, when the network connection is available. This means that messages to and from the centre may trip over each other. It is perfectly possible that, if not treated properly, messages from the centre may overwrite data held locally.”
>Four years later, former Fujitsu engineer Richard Roll wrote in a witness statement to the High Court: “The issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at branch level.”
BUT the most important part
>So far, nobody at the Post Office or Fujitsu has been held accountable
There’s barely any legal support at all these days. That’s what all that “tough on crime” and “stop waste” nonsense in newspapers gets you: large chunks of the criminal justice system barely work anymore.
> In the latest chapter of one of the biggest miscarriages of justice in English legal history, 39 people who were prosecuted
Meanwhile in the Netherlands, ~26000 people have been branded as fraudsters by the tax office due to a way too strict child benefits law. More than 100 probably entirely innocent people fled the country. Even the compensation that is now promised is only slowly trickling towards them, and likely to be snatched up by debt collectors - including even the tax office itself, which is still partly unrepentant. Okay, they haven't been sent to jail directly, but the scale of this is huge.
I'm sure it's a coincidence that Fujitsu was also heavily involved in the NHS IT fiasco which cost the NHS £10B. 'the biggest IT failure ever seen'. The Fujitsu UK chairman is also a large Conservative party donor of course - also a complete coincidence. https://www.vice.com/en/article/59x7wz/fujitsu-uk-sues-depar...
The link out to another story[1] has some interesting details...
"In December 2019, at the end of a long-running series of civil cases, the Post Office agreed to settle with 555 claimants."
So settlements in 555 of the original 700+ prosecutions.
"It accepted it had previously "got things wrong in [its] dealings with a number of postmasters", and agreed to pay £58m in damages. The claimants received a share of £12m, after legal fees were paid."
But 80% of the settlement money went to lawyers. Ugh.
The percentage isn't the problem. The problem is of the settlement amount doesn't include damages and also legal fees, both of which should be the responsibility of the perpetrators.
I disagree. Even the ambulance chasers here in the U.S. take around 40% as their contingency fee. 80% is just...wow.
Edit: "ambulance chasers" in this context means very opportunistic lawyers that are primarily motivated by money, and not helping their clients. I don't see how that term is disparaging any victims/clients. The comparison is that even outright greedy lawyers aren't taking half+ of the settlement. In this case, using £250/hr, the lawyers spent 88 lawyer years worth of time (184k hours).
It was an extremely complex case which was very hard to prove, against companies which belong to the establishment and had been shown the benefit of the doubt by the legal system on multiple occasions.
You're comparing apples to oranges.
"Ambulance chasers" (a terrible slur that looks down on weak victims pursuing justice), offer their services in a competitive market. If they charge too much, again, that should be determined by having a separate pool for fees separate from damages, and be a dispute between the perpetrator and the lawyer, not the victim and the lawyer.
The cost of the legal work is uncorrelated to the size of the damages.
Limiting legal fees just makes it not cost effective to pursue justice for smaller damages with more complex cases.
It's absurd bordering on evil to say the problem here is that people got paid too much for their excellent work (fighting against the resources of a corrupt major corporation and a corrupt major world government!) not that the perpetrators was under punished for their horrific crime.
The heroes who saved 700 people's lives deserve the money more than super-wealthy psychopathic perpetrators.
Then why not give them 99.9% of the take if they are such big heroes?
Because for the lawyers to get all the money each time harm happens means they more from harm to people than the people themselves benefit, this is a perverse incentive to keep the system exactly as it is for people who often become our lawmakers.
This response bears no relation to the topic at hand. As said earlier, the damages and the legal fees are two separate things that shout be kept separate.
> "Ambulance chasers" (a terrible slur that looks down on weak victims pursuing justice), offer their services in a competitive market.
I wonder how other countries get by without "ambulance chasers". The only country I know that has them is the US, and their existence is the sign that something is fundamentally wrong.
There's obviously a lot of detail there, but it does still feel to me like more than £12M should have gone to the actual post workers. That's ~22k each.
An (IMO) Interesting question is how to reduce the risks of things like this happening.
Where evidence from IT systems is being used as a large part of a prosecution, it seems that it should have some kind of scrutiny as to how those systems operate.
One option would be allowing the defence to see details of how the system works, testing that was done and known bugs, but that would require a lot of expensive work by legal defence teams, especially where the system is complex.
Another option would be some kind of certification of IT system operation, but again it would be hard/expensive to do and very incompatible with rapid development techniques.
I'm very sure this system was certified in a multitude of ways. No certification process would prevent this.
The real issue here was that Post Office refused to recognise that, although computers themselves are mostly infallible, computer programs are never infallible. They conducted their activities and took actions based on assuming the reporting was flawless.
Then the really serious problem is that in cases where the fallibility became more visible, they consistently and systematically covered it up and pressed forward with their incredibly aggressive enforcement work anyway, knowing how much damage it was doing.
This is unquestionably an issue of abuse of power and position.
> although computers themselves are mostly infallible
What do you mean? Hardware is fallible too, just less often than software. This may cause problem on its own e. g. bit flips in non-ECC memory, HDD which lie (reply to flush cache before data is actually written) or HW can trigger software errors, e. g. HW can crash at random moment and SW can be not designed to handle this properly.
> An (IMO) Interesting question is how to reduce the risks of things like this happening.
I look forward to finding out if this was a “fraud system gone wrong” or a more basic ledger system failing to do sums correctly.
Partially addressing your question though, if you were to insert the words “AI” and “bias” into the sentence we as an industry are starting to figure this out. The certification and testing processes you mentioned are there in cases where a team’s mature enough to have both a data and model lifecycle worked out. You see words like MLOps trying to describe how to do that effectively in production.
For example, my work has both a design approach (in both the product design touchy/feely sense and software architecture sense) that includes questions and practices that will help to reason through data needed to address a problem, what can go wrong with that, and how things look when it goes wrong. The last bit is the most interesting one to me. In terms of practical engineering, inference results generally should have some sense of lineage - of data, model, and training services which explain how you got to a given answer, including what inputs were considered or ignored.
An interesting side topic with this is that poor implementations can result in inexcusable differences that affect downstream systems. For example, if a particular model has predicted something like “this transaction is suspected to be fraud” it better be consistent from run to run, and the input data better be consistent over time. If either of those changed - explaining that to the consumers of the data is essential to them understanding that either the model changed, the data changed, or both.
>An (IMO) Interesting question is how to reduce the risks of things like this happening.
Corroborating evidence. In this case, where was the evidence that this money was ever in their possession? Was it ever sitting in their bank account? Was it buried in the back yard? Did they buy fancy sports cars or houses? The prospect of thousands of people stealing money without a trace of the cash is fantastical.
In general, I'd say electronic evidence should need to be corroborated with physical or other types of evidence to achieve a conviction. It's too easy for electronic records to be falsified, either through software bugs or outright malicious intent.
Presumably to put someone in prison for being a money thief, one would need to prove where that money went...
Were all these people accused of theft with not a single record of the yachts they bought with all the money they supposedly stole?
I would assume most of these people would be able to turn over a complete financial record of their lives (ie. I was paid £x, I paid taxes of £y, and here is a bank statement showing how I spent it, and here is whats leftover). How exactly can you imprison someone for theft of money if they can present that?
Post Offices handle a large amount of cash, much more than any other business of their size. Many of the sub-post offices in question would be paying out pensions and welfare benefits in cash to a large proportion of local customers. If someone was stealing from the post office, they could easily do so in cash.
So 500 peiple stole millions and the prosecution cannot show where a single penny went, noone even got a new car or TV? Did they eat the money?
And all the evodence the prosecution has are electronic records, entirely in their control, which they could fake and which were never checked by a third party for basic errors? This is a colossal miscarriage of justice
Perhaps it is, but if there had been credible evidence of a theft 'I would not have been able to spend all that money in cash' is not the basis of a solid defense.
There have been genuine cases where accountants, bank managers, and so on have embezzled large sums of money, including in cash, and spent it all untraceably on things like feeding a gambling addiction.
Unfortunately people died during this time too and did so with this hanging over their head. An absolute scandal, but there's no inquiry into the directors involved. Not yet, at least.
I just came from another thread (here) where the subject was Google arbitrarily ruining businesses and lives based on algorithmic fraud detection gone wrong. I'd estimate the issue is alive with U.S. techies at the very least.
Post Office told the post masters that they were short, so many post masters made up the shortfall from their own pocket, expecting the books to eventually balance and to get repaid.
When people were unable to continue making up that shortfall this was seen as further evidence of their criminality: "they've spent the money", "they've hidden the money", and not "they never had the money".
It’s insane they stopped pulling the thread and the defence didn’t push that, even if you spent it there would be evidence. I would have been highly skeptical that all of it just disappeared Into thin air across the entire group....nuts
Pentesting and auditing aren't great solutions here. They can be useful on small scopes but a big system like this, it's unlikely to be hugely impactful – it will find things, but who knows if it finds enough.
In the UK in the wake of the 2008 banking crisis, a number of positions in banks became criminally liable for issues under them. If you're director-level or above (I think?) then you may be ultimately put in prison for negligence or issues like that which occur in your department. This is rare, not sure if it's been used yet, but it effected a cultural change in consumer banking as a bunch of execs suddenly had their necks on the line if someone under them did something wrong. I don't believe this is too hard-line in practice, I think a defence is "look at all these reasonable steps we take, we couldn't have foreseen this", but it had the impact (source, a good friend of mine is bordering on this level in a UK bank).
I wonder if a similar thing could work in a wider way across more industries - not with the intention of criminally punishing lots of people, but with the aim to change the culture around responsibility to the public and other stakeholders in the work that we do.
Not taking software results as a fact. Software report stating X in court should be equivalent to "the person who wrote this in a hurry would say X, but it's not a sworn testimony".
We should have the person presenting any report like that be personally responsible for the contents. If they aren't willing, it shouldn't be presented.
We should have the person presenting any report like that be personally responsible for the contents. If they aren't willing, it shouldn't be presented.
I don't think making it personal works at scale. You can't reasonably expect everyone giving evidence in court, say every individual police officer who is a witness to a speeding offence, to be a technical expert on the technological tools they are given to do their job.
Instead, as you implied in the previous paragraph, the weight given to any evidence derived from technology should be proportionate to the credibility of that technology. If it's a device that has to be vetted and approved according to strict regulatory standards and in court there are two other concurring sources of evidence, that's clearly a much stronger case than a single reading from a single device whose calibration has reasonably been called into question at trial that is being presented as the only evidence in that trial.
> say every individual police officer who is a witness to a speeding offence, to be a technical expert on the technological tools they are given to do their job.
That's what I was going for. If the officer doesn't understand the limitations of their tool, they shouldn't testify in court beyond "I pointed it that way and read the number, as trained".
There are existing cases where the speed reading is contested because the handheld speed cameras can move slightly and bounce first off the side mirror then off the reg plate giving you "extra speed".
My point was that if you say "that person was speeding" you should be responsible for that statement afterwards, but you can say "I used the provided tool and got reading X", at least the doubt is there.
FWIW, I'm reasonably sure that's exactly what does normally happen in that particular case. Police officers sometimes speak in a slightly stilted way in court here in the UK, partly because they use words carefully chosen to be statements of fact as they know them and not to draw conclusions that are a matter for the court to decide.
It's not about positive incentives, it's about the lack of negative incentives. More true negative incentives need to be shifted onto the production side, back onto the corporations, its officers, its middle management, and if required down to the individual contributor.
Corporate structure helps diffuse and deflect responsibility. Each group (executive leadership, middle management, and ICs) gets to diffuse and deflect responsibility and liability onto each other.
We already have all the positive incentives in the world - cash money. It's not enough.
Standards. Just say certain things, payment systems, need to meet certain levels of auditability (does it record all relevant data, and can I see them after the fact), verification (is the data correct and can I prove that) and privacy.
A shocking injustice. Innocent people went to prison for years. There was clearly a cover up at Fujitsu and the Post Office, and those accountable should now be prosecuted.
Unfair but as a spectator so frustratingly lacking any proper answers. It seems nobody could ever even work out if any money was missing or not. Let alone why. No closure. Just official judgement that no one knows...
BBC radio 4 did a thing about this, even when the post office knew they kept going throwing people in prison. It's so depressing. What's also depressing is that people trusted this software. How did the defence teams never question it properly the first time. I mean if it's a ledger, prove it works.
I hadn't heard about the story until the BBC started re-running this series this week. Absolutely shocking that flaws in the system were dismissed and suspicion thrown on the sub-postmasters instead.
As a software person, I would like to read a more detailed post-mortem on the issue from a code, engineering and project management point of view: e.g. who built this software, when, with what process, with what safeguards, and how did they fail? Was it in-house or outsourced, and if so, to who? Did it run on-premises? What checksums, what backups? What lessons, if any, did they learn and what can we learn?
In fact, the more I think about it, the more I think that a forensic post-mortem with lessons learned and changes made should not be optional: After an air disaster, it would be mandatory .And this really was a software equivalent of a plane crash.
“It is hard to imagine a more stupid or more dangerous way of making decisions than by putting those decisions in the hands of people who pay no price for being wrong.” – Thomas Sowell
The phrase "Affront to justice" is key here. To be honest, I am completely shocked that this wasn't sorted out several years ago when it was all over the papers and it was completely obvious what had happened. But that key phrase allows the wholesale claiming of damages.
It's also noteworthy that these injustices originated from private prosecutions brought by the Post Office. That is a relatively unusual legal action in this country, where almost all criminal prosecutions are brought by the state. Given the damage that a wrongful criminal prosecution can cause, including imprisonment and having a criminal record, the compensation awarded could be considerable and there is already talk of the Post Office needing extra government funding to cover the cost.
Another small point of interest that doesn't seem to be making the mainstream reporting yet is that under our legal system the state prosecutor (the Crown Prosecution Service) has the power to take over and, if appropriate, shut down any private prosecution. When the inevitable inquiries publish their conclusions, the fact that so many bad prosecutions were successfully brought over such a long period might reflect poorly not only on the Post Office and on the courts and lawyers involved in the convictions but also on the CPS for not intervening. This could become politically significant, because the current Leader of the Opposition was in charge of the CPS around 2009-2013, the last five years when most such prosecutions were being brought. That could leave him in an awkward position if he's attacked over his record during the next general election campaign, given that his party is exactly the one that's supposed to stand up for working class "little guys" like the victims in these cases.
The company I work for ships hundreds of packages through RM. The RM tech I've seen is a mess. Makes me wonder what it's like behind the scenes. Just one lowlight I've come across, this comment can be found in the HTML for one of their portals:
One nitpick: Royal Mail and Post Office are two separate companies with independent boards. Royal Mail is the network and carrier, while the Post Office is the primary entry point into that network (they also offer access to a bunch of other services not related to the Royal Mail). Doesn't make your point any less valid, but wanted to call out the distinction.
As a sub-nitpick, I would definitely say Royal Mail itself is certainly the primary entry point into the network too. But Post Office is super helpful in providing supporting services for many government-related things like passport photos, certification, applying for things etc
Sounds like the kind of bullshit BT and OpenReach pull too. Claim to be two unrelated companies and yet one owns parts of the other and the same boards run both - all so they can pass customer problems between the two I definitely.
I'm honestly surprised anyone complains about that split - it was introduced specifically so that BT wouldn't control the entire telecom infrastructure in this country, and OpenReach was formed to provide equal access to all operators - BT being only one of them. This is an extremely good solution to what used to be a massive inequality problem previously. So no, BT and Openreach aren't split for some bullshit reason, they were ordered by the court to split in order to protect consumer rights and increase competition, goals which were overwhelmingly achieved due to that split.
And yes, the negative side is that every time something goes wrong, BT really can't fix it any faster, it's all down to OpenReach to maintain the network. But on the other hand, it always goes through OpenReach, whether you are with TalkTalk, BT or Sky, so the entity responsible for maintaing the network isn't the entity selling you broadband for home.
It would be an extremely good solution if it worked as intended. In fact Openreach were not fully independent from BT for most of their existence, and they operated the network in a way which was extremely favorable to BT for a long time.
Thus, the two companies extracted an exorbitant rent for the formerly public goods they controlled. The fact that some of this rent went to inefficiencies of running two separate companies on an illusionary arm's length basis does not really improve matters.
This sounds like a similar situation to Telstra in Australia, which was forced to split into two entities - one a wholesale network provider that was open to all operators, the other a consumer operator that (supposedly) operates under the same rules as everyone else.
Openreach
Mike McTighe Chairman
Clive Selley CEO
Matt Davies Chief Finance Officer
Edward Astle Non-executive Board member
Liz Benison Non-executive Board member
Andrew Barron Non-executive Board member
Jon Furmston Secretary to the board
Simon Lowth BT Group nominee
BT
Jan du Plessis Chairman
Philip Jansen Chief Executive
Simon Lowth Group Chief Financial Officer
Adel Al-Saleh Non-independent, non-executive director
Sir Ian Cheshire Independent non-executive director
Iain Conn Senior independent director and independent non-executive director
Isabel Hudson Independent non-executive director
Mike Inglis Independent non-executive director
Matthew Key Independent non-executive director
Allison Kirkby Independent non-executive director
Leena Nair Independent non-executive director
Sara Weller Independent non-executive director
Rachel Canham Company Secretary & General Counsel, Governance
One nitpick: Royal Mail and Post Office are two separate companies with independent boards.
Though as a nitpick of your nitpick, they weren't truly independent until the relevant provisions of the Postal Services Act 2011 came into effect on 1 April 2012. What we know today as the "Post Office" and "Royal Mail" had a long history before that.
I think you're possibly confusing the Royal Mail with the Post Office, there. You're talking about the Royal Mail. This article is about the Post Office.
The idea that 700 people in the same job were all committing the same crime and constantly getting caught is insane. This is a perfect example of Orwell's description of fascist Britain, where the people are made slaves of the state.
Over several years and 20,000 post office branches it's not a huge percentage. I suppose they assumed the new system was revealing corruption that had gone unnoticed under the previous system. That's in no way an excuse or justification for the knowing, deliberate suppression of evidence that went on here.
> the convictions of 39 former postmasters ... the UK's most widespread miscarriage of justice.
There's no way this is true.
> There were more than 700 prosecutions based on Horizon evidence. The commission and the Post Office are asking anyone else who believes their conviction to be unsafe to come forward.
On second thought, I guess it may be, since even after the abuse was proven they are still holding innocent people on false charges.
>"since even after the abuse was proven they are still holding innocent people on false charges."
Well, same government first destroyed immigration papers and then deported and otherwise ruined the lives of their own citizens ( Windrush scandal ). I'd love to see the perpetrators in jail but fat chance.
The Real travesty here is that people can't afford to pay for their lawyers (let alone a software expert or QA to actually look at the code or test it) , they aren't entitled to representation, so they have no option but to plead guilty.
Repeating this comment that I posted yesterday...it is unfair?
Will any developers involved in this horrible scandal ever will be held accountable for their work?
I wonder if the developers who were responsible for such a bug-infested piece of software realise their work has destroyed people's lives? (They presumably never met the users of their software or were so distant from end-users that they never considered the consequences of their actions.)
Do those developers even realise it was their incompetence that caused untold misery? Or are they completely detached from the events in this scandal and see themselves as simply cogs in the 'system' and thus blameless?
Blame must be apportioned to management. But also I feel it's too easy as a developer to see yourself as part of a team and thus absolved of any individual blame. You're subsumed in the "team" - and ultimately no-one takes responsibly.
Even with management at fault, one cannot deny that it was the developers who produced absolute garbage.
I hope the developers who worked on this system, no matter how much they feel they are not responsible for the failure of this project, will reflect on how the impact of software they built had devastating consequences on people's lives.
When ATMs were introduced in Canada in the 70s/80s, it was common to believe they were infallible. When customers claimed they were short-changed by machines, often they were prosecuted for fraud or attempted theft.
I'm sure HNers can think of dozens of ways a machine could be wrong ...
Also, regarding the Postmaster article, note that somebody working on that project would likely face great difficulty in convincing anybody there was a systems problem.
It's like a murderer giving evidence against a random stranger and being believed at face value because they provided all the evidence first hand.