Nope, definitely systemic within the company. The engine is a system that must meet certain requirements. The standard way of meeting those requirements is to have a urea injection system. That's what the competitors in the US do; it's also what Subaru, for example, don't do, and they don't sell into the US market.
In order to pull off this little 2-person hack that Horn alleges, those couple of software engineers would have to have a major say over the entire engine design. I call bullshit. This little hack ONLY makes sense in the absence of other major engine components.
The company culture is clearly absolutely rotten and their execs are liars. Employee morale, especially in engineering, must be through the floor.
> those couple of software engineers would have to have a major say over the entire engine design. I call bullshit.
I agree. Developers wouldn't take that amount of corporate risk on for such a hack without having significant backing from management.
Even if they were completely on their own, which is impossible for me to believe, at some point another developer would have noticed and said something. Developers are smart enough to know anything that can go wrong will go wrong and that such a hack would be exposed, taking a lot of people down with it.
The "individuals" VW is claiming are responsible are 10 managers and 3 top engineers, which include the head of R&D, the head of engine design and the head of Audi.
The definition of "individual" might be different between what VW claims and what you read, though.
Michael Horn, CEO of VW America: "This was a couple of software engineers who put this in for whatever reason." That sounds nothing like what you're describing. The article alludes that many were suspended, but it doesn't directly attribute blame to them.
I find it impossible to believe any software engineer at the implementation level would assume that level of risk. This is obviously scapegoating. I really hope they set those engineers up for life, because they'll never be able to find another job again.
Well, I’m extremely disappointed with VW US. VW Germany instantly had CEO resigning and taking full responsibility and in days started suspending top managers.
In contrast, VW US seems to be even more a bunch of liars.
Well, he's kind of a dead card anyways, so why not use him through all the Congressional hearings. Replace him with someone else when the blizzard is over.
Isn't that sort of like a company president blaming "unauthorized excesses by employees" while neglecting to mention that the "employees" were actually all VP-level?
It sounds to me like they're either casting the blame on the rank-and-file, or they're heavily bending the truth by trying to make it seem like the culprits were "merely" regular employees.
It's also what another VWAG member, Audi, does on the TDI's in America.
These were also very high level engineers.
I can also totally believe that, in order to not seem like a they wasted billions on engine development that they literally would not be able to sell without significant changes, that people in charge of the project would just figure out a way to make it seem like a success.
Contrary to what you say, this doesn't require an entire corporate culture be rotten. It's like branches of a tree. The fact that one branch is diseased does not mean the entire tree is bad.
I've seen this happen in other companies before just in my little open source licensing world.
(some group head and his group lie to me about what they are doing, skirt the rules, etc. right up until i catch them. His boss then asks him what the fuck he was thinking and demotes or fires the group head)
As you get higher up, one of your biggest problems is, fact, people don't want to tell you bad things. My VP used to have a sign on his desk that said "no surprises". Because he wanted to know everything, whether it was good or bad.
"Branches on a tree" makes more sense if VW didn't drag their feet for a year and a half when the report of the failing vehicles first came out. I agree that the whole company cannot be rotten, because it would have leaked to the public long before now, but upper management may be far too ok with rules skirting.
Look into the VW / Porsche infinite share squeeze. These people aren't affable fools. They didn't just accidentally let engineers write code to skirt around international rules by a factor of 10-40.
The current scandal is only about the "EA189" engine family with NOx traps for exhaust treatment. The new "EA288" engine family with urea injection was designed from scratch, and, I hope, doesn't need cheating software.
Edit: the EA288 is/was -- at least in Europe -- also sold with NOx trap, but VW still claims it's free from cheating software.
Considering all we've learned in this saga about the industry in general and Volkswagen in particular, I would say it's more than likely that it had cheating software anyway.
I think this is part of the problem. The EA189 without AdBlue needed the cheating software. The Passats with AdBlue also had the EA189 and likely could have been tuned to work without the cheating software, but likely have it anyway.
It's a bit of a shame that EA288 cars are stacking up at dealers nationwide and will apparently never be certified for sale (VW said they would not ask for certification), even though they almost certainly don't need the cheat. (Although it's possible that it's still present, which is of course still illegal, need it or not).
Having a lot of say in design, and keeping secret an intentional violation of government regulations from your entire international company are two pretty different things.
I'm making assumptions, and I believe this story is pure, unadulterated bullshit from top to bottom.
Is it likely that the heads of R&D and engine development at VW are hands-on and writing code themselves? No, its extremely unlikely. Having worked in several large companies, there are 0 people at that level of management that perform hands-on work.
It is likely that those particular positions benefit and are motivated to cheat on regulations for the benefit of the company? No, the bottom line of the company is affected, and it is far more likely that business people are applying pressure.
Is it likely that they could make those changes in a company that size, without anyone noticing? No, not really. Software in a large organization leaves trails everywhere and requires explanation, they had to interface with other systems, and people had to interface with theirs.
Is it likely that the same narrative structure that our government attempted to use to excuse torture, is something VW is also trying? "Oh, we had no idea, it was one deranged guy who did this." Yes, it is highly likely that prominent executives are scrambling to get out of the way, and highly likely that the company is trying to come up with a story that makes it look less corrupt than it is.
> It is likely that those particular positions benefit and are motivated to cheat on regulations for the benefit of the company? No, the bottom line of the company is affected, and it is far more likely that business people are applying pressure.
I agree with most of your comment, but not with this point. Being the Head of R&D who is able to boast about bringing a "breakthrough" new diesel engine to market is bound to come with some personal perks.
You can re-define this as "business people are applying pressure" if you like, but that does not change the responsibility of those position - and besides, it's kind of a cop-out. With that thinking, you can re-define everything as being down to market pressure.
Somebody made a decision to do this, and I can easily see an unethical head of R&D making that call for personal career reasons - just like unethical researchers cheat by making up data. I'm not saying that that's necessarily how it happened, but it's a definite possibility.
The thing is, you are currently defending the prominent executives.
We’re talking about the heads of R&D and the head of engine development – both more business people than engineer.
Additionally, 2 CEOs have resigned over this (in just 6 months!)
And it’s known that the first of these CEOs tried to focus on engines without Urea injection – which did get 280mpg, though – because he personally was invested into their development and tried to push the heads of R&D and Engine development to do this.
That’s why the CEO originally responsible even resigned before it became public. And why the second CEO resigned: because, despite not believing he himself did it, taking the responsibility for it.
“Having a lot of say in design, and keeping secret an intentional violation of government regulations from your entire international company are two pretty different things.“
But the difference between intentional benchmark optimization (which the whole industry is doing) and illegal certification manipulation is a rather subtle one. The former would a perfect front for the latter and the intersection of all employees who sufficiently understand the legal situation and those who sufficiently understand the engine could theoretically even be the empty set. (in which case blame would default to those high enough in the org-chart to have members of both groups under them)
The two heads who lead the R&D and engine development are not software engineers. But they would certainly require software engineers to help them carry out the deception. I highly doubt that only 2 people are involved.
perhaps 2 people who were intentionally intending to mislead. the software engineers could have been told the software was for factory testing only and it would have been their job to produce it. The actual program itself was not illegal just the fact that it was used to mislead official testing.
This ain't no 'independent contractors on the Death Star' issue here.
Not only that but by finding themselves a scapegoat, management has ensured that in the future, honest mistakes will go completely unreported. These execs are clearly more interested in CYA than actual results.
Also, although it's not proof, the simple "follow the money" principle should hint at the truth here. I can't see what 2 engineers would gain from putting so much time and effort into independently creating "defeat devices". In comparison, VW had a lot to gain from fudging the figures (if they hadn't been caught).
If I were an engineer at VW I'd be hunting for a new job right now. The company seems to have no problems pinning systemic failure on individual scapegoats, and I wouldn't want to hang around to be caught in the next round of ritual sacrifice.
"This was a couple of software engineers who put this in for whatever reason," - I'm not buying it. What would be their motive compared say to VW's motive? VW has a heck of a lot more motive for falsifying emissions than do "a couple of software engineers." Even if you were to argue the engineers were somehow fearful for their jobs that's still VW's problem because they're the ones who created that work environment instilling that fear.
I really expected VW to do better in handling this. Some business analysts are saying VW may be out of business within five years time. Based off this testimony I say good riddance.
No way could it be just a couple of software engineers. The engine is a system, and its the engine that has to pass the emissions tests. So that means the whole engine development team had to be in on it, including their technical management. They in turn would not have done such a thing if they had not believed that senior management wanted it, and it is the responsibility of senior management to make it clear what they want.
When this CEO of a company with 600k employees says "it's some software engineers", I assume he might actually be honestly referring to what we would call engineering managers, perhaps with hundreds of indirect reports, 2-3 levels down from the CEO.
I find this plausible...
(Particularly since these hearings seem designed to score political points rather than to find out what actually happened.)
The (now resigned) CEO prided himself on being an engineer and in depth knowledge. I don't believe he didn't know how VW was able to market clean diesel engines in the US without having a clean diesel emission system for them.
His big thing was to be the largest car maker and he needed to juice US sales to get there. He had to have spent a lot of time on US strategy. The shitty part is that he succeeded--they became #1 (this year!) and he has a huge pension to retire on.
"Winterkorn’s professional career began in 1977 as a specialist assistant in the research division "Process Engineering" at Robert Bosch GmbH. From 1978 to 1981 he headed the refrigerant compressor development group "Substances and Processes" at Robert Bosch GmbH and Bosch-Siemens-Hausgeräte GmbH.
In 1981 Winterkorn joined AUDI AG as assistant to the Member of the Board for Quality Assurance. Two years later, he assumed responsibility for "Measuring Technology/Sampling and Test Laboratory" at Audi. At the beginning of 1988, he was made departmental head of "Central Quality Assurance", and in 1990 Head of Audi Quality Assurance.
In 1993 Winterkorn became Head of "Group Quality Assurance" at Volkswagen AG and was appointed General Representative of Volkswagen AG in March 1994."
These would be his formative years. It seems he focused on processes and quality rather than actual technology, IMO.
fact is the whole report chain is rotten. who wrote the code, who told him so, and indirectly most engineers that had a reasonable knowledge to be in a position of suspecting a foul play.
you can't get this running for 6 years without widespread cooperation, or at least willfully turning a blind eye.
those engines ran loads of miles on benches before being put into cars, and it's a large group running multiple cross tests on many parameters including fuel consumption and efficiency and operating temperature, which should give anyone within the sector a good rough idea of how an engine performs emission wise.
However plausible it may be, it is still not excusable.
The company should have processes in place that can verify the claims it makes. Plausible deniability is really just a claim of neglect.
Edit: I'm being rate-limited, so I'll respond to the points below right here:
>What processes should be in place, exactly, to do what?
How about driving a vehicle in the manner most of their customer's do and examining the emissions?
>Quarterly polygraph tests performed by all engineering managers
How would that help exactly? Polygraph is demonstrably unreliable.
>to assure top management they haven't broken any laws?
It's mostly irrelevant how assured top management is about anything. What matters, in this case, is how far out of spec the vehicles Volkswagon produced are. That is what needed to be verified: specs vs. facts.
Let's assume the individial he is talking about is a manager of a group of hundreds of engineers working in this area. This means this guy's direct manager most likely would not be in a position to inspect the work of people 2-3 levels down for any irregularities.
What processes should be in place, exactly, to do what? Quarterly polygraph tests performed by all engineering managers to assure top management they haven't broken any laws?
All you need are strong whistleblowing protections and mandatory independent investigation of any claims made.
Do you really think of the dozens of people who had to be aware of it none of them would have thought "this is really dodgy" and wanted to push it up the chain?
At the end of the day, that manager is still responsible for everything that happens under them. That's the justification for giving them the big paychecks, isn't it?
Exactly. The most devastating part of all of this for me (a US citizen) is that the German CEO resigns in disgrace and there are ongoing criminal investigations in Germany, but in the USA the CEO holds fast and blames engineers.
It seems like the German government is more competent that the US at enforcing laws against corporations, even when those laws are USA laws...
*EDIT: This comment is poorly worded. What I mean to say is, Germany is doing a better job at holding PEOPLE AT THE TOP responsible, whereas the USA is predictably allowing CxOs to plead ignorance and hide behind the corporate veil. Meanwhile in Germany, http://www.wsj.com/articles/german-prosecutors-open-investig...
You realize this is the exact opposite of what the rest of the world is saying, right? Is this the first article you've read on the matter? I've been following it with great interest, and many others have expressed shame that it's the US that is catching a European company cheating, and threatening prosecution.
This all happened because of the work of the West Virginia lab, plus CARB and EPA's investigation, and the EPA's refusal to issue an emissions certification to 2016 diesel VW passenger cars which has led to likely tens of thousands of un-sellable cars sitting on US dealer lots.
The German government has been shown to have looked the other way repeatedly and to be complicit in fighting on behalf of the automakers instead of the environment.
The European emissions and mileage tests have been shown to be an utter and complete sham.
The story you're peddling is really the exact opposite of reality. Take a look around at the other stories that have been posted on HN on the matter.
actually the european started this asking for help since they had the equipement for static test, but not the equipement to fool the defeating device into thinking it was actually on a road.
Good call. It wasn't European governments, and it wasn't an attempt to punish automakers for malfeasance, though.
The ICCT (an NGO) that appears to be based on Europe, commissioned the West Virginia study to try to figure out how more stringent US Diesel standards were being met, so they could encourage automakers to meet the same standards in Europe.
"In 2013, his organization [the ICCT] commissioned a study of VW diesels by West Virginia University after questions were raised about European diesel emissions standards and whether European vehicles were emitting too much nitrogen oxides linked to smog. Testers looked at three diesel cars for the U.S. market: a 2012 VW Jetta, a 2013 VW Passat and a BMW X5.
The group expected the cars they tested would perform better than those in Europe, because U.S. regulations are tougher. They were surprised the two Volkswagens had significantly higher-than-expected emissions, while the BMW performed well. They were so certain they had done something wrong that they tested the cars two more times with similar results."
This was an engineering problem. There may be an assumption that while there are about a dozen people in the US who work for VW in an engineering capacity, the US part of the group is in essence purely a marketing operation.
From what I know of the car industry, coding standards are very low. I don't think any car companies would look good if their code was audited by professional software engineers in other fields.
Then you don't know the motor industry very much as it has very strict coding standards like ISO 26262, the code has to be audited internally and by certified 3rd parties with every release to meet the development cycle requirements of the standards.
The industry also has specific coding standard for every language that is used in embedded systems like MISRA-C
https://en.wikipedia.org/wiki/MISRA_C
Today every thing you do with your car from using breaks to deploying air-bags is done through the computer, there's no way in hell that the motor industry would be releasing poor code "intentionally" because it would cost them billions in liabilities.
And you're not familiar with what happened when a 3rd party (Michael Barr) was allowed to FINALLY look inside Toyota's engine code and determine the cause of the "Unintended Acceleration" problem:
"Barr checked the [Toyota] source code against MISRA’s 2004 edition and found 81,514 violations."
The model year 2005 Camry in question from the trial would have been made available in 2004, and the electronic throttle control system was designed prior to 2004 (IIRC, the ETC design was carried forward from the previous model). How does it makes sense to apply MISRA-C:2004 in this case, especially when the Toyota ETC in question was admittedly not designed under MISRA-C:2004 nor prior MISRA-C revisions?
It is inaccurate to claim Michael Barr determined the cause of Toyota's unintended acceleration: he proposed a possible failure mode that was persuasive to a lay jury. His proposed failure mode didn't leave a DTC, so there's no way to actually know if his proposed failure mode actually happened in this case (or in any other).
I remember when this was posted on HN before, what I always wondered about is the discrepancy between the violations that NASA found and Barr.
NASA found 7000 and change Barr found over 81,000, I'm not sure whats the size of the code that was reviewed but that's allot, even if you count the fact that half of them might be silly like naming convention violations that still leaves quite an odd number for actual violations especially when considering that the code for an embedded system can't be that huge.
And the 10,000 global variables thing, well 10,000 sounds like a huge number but there wasn't any mention if they were a) necessary, b) implemented correctly, and c) out of how many variables in total? if the code they've tested has say 1 million variables then well....
And how they described global variables is also weird, they claimed that every software within the system can access them to me this sounds that your stereo or the rain sensor for the wipers can override data for the breaks, but that might be true only if everything is running within a single application.
If the breaks run as an independent application well then only every function within the break software (depending on the language and how global variables are implemented) could potentially access those variables.
To me pretty much that entire explanation seem to be constructed to sound worse than what it is, which is why I think that expert witnesses should only be allowed to be appointed and called by the court it self (as in by the judge so they and the jury could understand the technical details better) and not by either the defense or the prosecution.
> there wasn't any mention if they were a) necessary,
They were not. It is never necessary to have 10,000 global variables. If you have 10,000 global variables you are systematically Doing It Wrong. (That's not to say it's uncommon. From my experience I strongly suspect that it's a very common practice.) As one of the witnesses put it, quite correctly IMO:
"And in practice, five, ten, okay, fine. 10,000, no, we're done. It is not safe, and I don't need to see all 10,000 global variables to know that that is a problem,” Koopman testified.
Trying to justify this is like trying to justify not having a bug tracker. Which, BTW, they didn't.
> b) implemented correctly,
There's no way to check 10,000 global variables to see if they're implemented correctly, and that's the entire point.
> and c) out of how many variables in total?
That is completely irrelevant.
> they claimed that every software within the system can access them to me this sounds that your stereo or the rain sensor for the wipers can override data for the breaks,
This was the ECU, it only controls the engine, not the radio or the wipers. It has tasks for stuff like monitoring the engine and wheel speeds, the accelerator and brake pedal positions and controlling the fuel injection. And all of those tasks were only an extra header file and a typo away from stomping on one of the 10,000 global variables belonging to another task, but that isn't even the issue. The issue is that there is no way to trace the flow of data in the system because it's completely unstructured.
> but that might be true only if everything is running within a single application.
Welcome to the world of embedded RTOSs, where everything is running within a single application. It's not a PC. "Applications" aren't a thing. There's certainly no memory protection.
> Welcome to the world of embedded RTOSs, where
> everything is running within a single application.
> There's certainly no memory protection.
Generally, I agree, even though there have been quite a few chips already that have a primitive "Memory Protection Unit" that typically only distinguishes two processor states and you'd be able to protect the working data of e.g. the scheduler and your watchdog from the rest of the code running on your system. Which is much better than nothing!
But in the end the important consequence is: Having everything in one address space encourages people to violate the concepts that memory protection between tasks on your PC typically enforces, e.g. tasks reading from, and writing to other tasks' data structures without proper synchronization; instead of using proper IPC mechanisms like queues. Which is probably what these 10'000 global variables were used for.
And 2 of my friends worked for BetterPlace (RIP) and it took years for Nissan to accept the code which had to be rewritten to comply and certified by an external 3rd party.
And that code wasn't even "mission critical" in terms of road safety.
I have seen both very good code and very bad code. Standards are in place for making it good but it really depends the most on who's implementing it and the SW team culture.
The code might even have been outsourced. I'm not trying to disparage outsourced developers, I know what I might do if the user story was something like "Needs to pass emissions when attached to test system."
It shouldn't matter. Another test process should have caught incorrect code. The car should be tested to the specs and the specs should reflect the regulations.
If that's the case, the execs are even more at fault for not having proper controls in place to prevent rogue employees from jeopardizing the entire company.
And, supposing this is true, would you actually be surprised by this? This is a car company, not a CI and code-review obsessed software company in Silicon Valley.
> This is a car company, not a CI and code-review obsessed software company in Silicon Valley.
I read that as:
> This is a company whose products are high-velocity 1500kg chunks of steel zooming around in public, not a company that sells an intangible virtual good/service
---
Yes, I know software is historically poor with car companies, but we should still expect better. Let's not just lower our standards because of cynicism :-)
I found products made by car companies to be in general more reliable than those made by software companies (either in Silicon Valley or not).
I trust my life to my car (software included) every day. I wouldn't do that with any software of my smartphone, not even the OS.
Doesn't it blow your mind that there are no "standards of engineering" or whatever for software? There's no licensure body for software engineers who build software running your car, and therefore no accountability on a personal level.
When an engineer builds a bridge, she has to personally sign off on the bridge, saying it's safe, and is risking not only her professional career, but I think she can also be jailed and held criminally liable if the bridge kills people due to negligence.
It blows my mind, at least, that no such thing exists for software.
The big problem with all those standards, frankly, is that you've gotta pay money to actually evaluate them.
There aren't any guarantees that they'll be useful, that they'll match the modern development processes in your language, that they'll fit your problem domain, etc.
Those standards are there primarily to make the publisher a buck--not to represent the codified wisdom of up-to-date practitioners in a field.
Until we've got a truly open-source standard for people to code against, we should stop wringing our hands about these things.
If you could have perfectly safe software, without requiring individual engineers to be licensed, would that be acceptable to you?
I'm not asking if you think such a thing would be possible or not - I'm asking if you would accept an alternate means of getting what I think we both want.
And a German one and from experience dealing with German companies software isn't regarded I the same light as what they would consider "proper engineering"
Then you don't know the motor industry very much as it has very strict coding standards like ISO 26262, the code has to be audited internally and by certified 3rd parties with every release to meet the development cycle requirements of the standards.
http://www.ldra.com/en/software-quality-test-tools/group/by-....
The industry also has specific coding standard for every language that is used in embedded systems like MISRA-C https://en.wikipedia.org/wiki/MISRA_C
Remember the Toyota code, or at least the review, that was leaked a while back? I don't find it surprising at all that it would be possible to sneak through some stuff that no one knows what it does. And in that kind of environment, people generally don't mess with stuff they don't know what it does, cause they don't know what it'll break.
Just because VW might have more motive to do this doesn't automatically mean that they're lying when they say it was the fault of individual engineers. Guilt is not determined by "who has the most motive", it's determined by who actually did it.
Now it may very well be true that VW is lying and it was a corporate decision. But it's not completely implausible that this really was the decision of a few individual employees.
Other commenters here are expressing surprise that engineers could "slip through arbitrary code" to do this. But isn't that kind of the job of software engineers, to write code to do things? I would hope they have code review going on at VW, but code review doesn't always catch everything, or maybe several engineers who were working together would review each other's code.
"But it's not completely implausible that this really was the decision of a few individual employees."
Oh, I fully believe it was the decision of a few individual employees. I just don't believe those individuals were the actual coders. Coders will code to the spec they are given; problems are far more likely to lie in the spec layer.
It is possible it will turn out the coders should have known better. But it isn't even that hard for me to create plausible circumstances in which the coders didn't have any reason to believe they were doing anything wrong. Remember, we're only seeing the end-results, but it's not hard to imagine a combination of specifications where the programmers are assured each little step is legal and OK, and a combination of other optimizations combining together to produce this result, with no one person quite understanding the interactions.
And just to be clear, I work a lot in security, on the defense side. I'm familiar with taking the possibility that coders will be actively hostile seriously. But I'm also very familiar with seeing five intelligent people each acting according to their own best info and with their own best interests at heart interacting to produce blithering stupidity. It's not always as simple as it looks when you're just looking at the outcome.
Like NASA, auto companies like VW produce a TON of documentation on all phases of design and engineering. This is especially true when it comes to regulation-sensitive processes like emissions testing.
As someone who has worked in a comparably highly regulated industry (pharma) I can state with full confidence that Mr Horn is a lying bastard.
> Like NASA, auto companies like VW produce a TON of documentation on all phases of design and engineering. This is especially true when it comes to regulation-sensitive processes like emissions testing.
Bingo! A lot of HN is acting like "it could slip through the processes we have at work so I can see how it could happen here" without realising that the processes, documentation, and independent review requirements are 100x more involved and thorough for critical car firmware than they are for a basic software shop.
Yes and no. Remember the Toyota firmware fracas and the bizarre coding practices (100s of global variables anyone?) that were eventually revealed. Adding heavyweight process doesn't necessarily solve more problems than it creates.
I'm not sure emissions controls are considered "critical firmware" in the same sense as systems that could actually kill the occupants of the vehicle if they fail. Or rather, I hope they aren't.
They will blame some small man to avoid prosecution of higher staff. I bet this was done off the records - so the Software Engineer manager have no proof this was ordered or suggested by upper management. At worst they will get 1-2 engineers arrested while they will move quietly to other companies that will be willing to use their experience and connections.
Of course. Fall guy 101. Not some tinfoilness, but expedient politics of surviving corporate death pits.
The sort of shops where most every discussion is double top-secret: happens in-person, quietly, behind closed doors and the email trail is kept to strictly vague coordination details, at most, since it's subject to discovery. Oh and information is hoarded, because details create weaknesses.
Arrested? Remember that cars don't pollute, drivers do or something like that. At the end of the day, what the engineers in the trenches did was design a device that is illegal to sell in some other country. It is correct that the blame goes to a place higher up, where the decision to sell was made. Which is what is actually happening at Volkswagen proper. The surprising part is that the VW USA CEO seems to be completely out of the loop about this.
Exactly. I don't buy this one bit. If those engineers feared for their jobs because the cars wouldn't pass, I bet those engineers could have left VW and found better jobs. Nobody needs that kind of stress.
This is probably a little reactionary, but can we just stop for a second and admire the difference in American CEO's versus his European counterpart? The latter resigns immediately as a way of accepting he is ultimately responsible, the former points fingers. This is a blatant example of why the tide of opinion is turning against those with the most wealth.
It's pretty close to the end. Fines will be levied against the Volkswagen company, not to him personally. His stock options will become worthless but he's still set for life. I think in this area, at least, China does it right. Shoot the guy as a warning to the others.
The Staatsanwaltschaft Braunschweig is leading the criminal investigation and they said specifically that they are not after Martin Winterkorn, the press reports.
The cars were designed in Germany, weren't they? (If not, excuse my shock to find we've been subjected to misleading advertising.) I doubt the engineers reported to the USA guy.
Wow, that's amazing. I would never have thought it was sensible to make all those tiny changes from one market to another. I guess it's kind of like the changes they make between model years...
Right. And those tests and regulations exist in the interest of public health, not product benchmarking. So the issue is bigger than lying about some performance metric to compare more favorable to competitors. And it's also not a domain that's auxiliary—this is engine technology, the very heart of the company and the product. And good god, they built an entire brand campaign around it.
I really do feel this is analogous to some large pharmaceutical finally admitting that they cheated clinical testing on an important drug. You can't just pin that back to a couple of "rouge scientists".
VW will not be out of business in 5 year. Nobody will remember this in 5 years. All the major automakers have manufacturing scandals in their past that resulted in actual deaths, not just fudging some arbitrary emissions standard. They are almost all still in business.
Probably, possibly, statistically contributing -- yeah maybe. But other automakers have made decisions that directly, indisputably, really killed people. Usually to save a couple of dollars on a part.
Even if you look at it from a corporate politics standpoint, a software engineer will much sooner say "car's not passing emissions? Not my problem -- take it back to engineering".
It's an utterly brazen lie to say the company didn't know about this.
Anyone who's ever built anything as part of a team - whether the Linux kernel, a space shuttle, or a Lego set - can tell you design decisions like this cannot be secreted into the final product.
So let's say a software guy decides to make this fix. How do you do that?
- You'll have a version control system, won't you? At the very least someone will see it. There's some interesting if statements referencing the speed of the rear wheels, which you can explain at code review.
- How do you test it? I doubt your normal office happens to contain the entire running setup of a diesel engine, with nobody around to notice, and nobody needed to tell you about the details. Because you know, an engine test rig wouldn't have details, right? You just turn it on yourself, spin the car up to speed and debug, right? And I'm sure there's nothing complicated about gas measurement, even though it tends not to be a part of CS courses.
- What do you tell the cluelessly naive good guy engineer about his results? Would he really only test the car under test conditions? Does any engineer with even the slightest grain of curiosity test only the one mode that he's supposed to?
> What do you tell the cluelessly naive good guy engineer about his results? Would he really only test the car under test conditions? Does any engineer with even the slightest grain of curiosity test only the one mode that he's supposed to?
While I agree with the rest of your posts, I can't let this one slide. This happens ALL THE TIME. I have to beat engineers senseless over this on a regular basis.
"Just because it passed one test, one time, does not mean you are done. It has to do something sane even when the rest of the world is coming flaming down around it. Crashing could be the sane thing to do; logging could be the sane thing to do; returning very obviously insane values is non-ideal, but might be your only option. A couple of safety checks are warranted."
I guess they are worried about significant legal action against the top-brass and are throwing off some chaff. Will the legal system go after the little guy again or land the big fish?
>design decisions like this cannot be secreted into the final product.
I'm not entirely convinced. You compare it to the Linux kernel. I'd say a better comparison is router firmware upgrades.
If a manufacturing plant gets sent a new image with a note attached saying "HQ says use this one" then they're definitely going to load it onto the cars.
Clearly someone cooked the image somewhere & there is likely to be version control in place but I doubt the whole thing is as transparent code-wise as you suggest.
How is Volkswagen's US head, who lives and works in the US, able to issue proclamations with such certainty about the non-existence of private conversations and directions taking place among the engineering staff and management in Germany, where he spends almost no time?
It is as if a US diplomat stationed in New Zealand used his authority to declare that Bush had never authorized any torture, and it was rogue individual interrogators who did it. How would he know given he was on the other side of the world when the conversations would have taken place.
I would assume he is speaking on behalf of Volkswagen, not testifying as an individual on his own experiences. Volkswagen's current story is that it was a couple rogue engineers, so he is conveying that. Congress isn't going to get the CEO in to testify, so it's reasonable to assume he is speaking as a proxy.
Exactly. If Mr Horn is certain of what he says, where's the proof? To make such allegations, any court would require that VW produce historical evidence of this misbehavior. Yet he offers none.
In the absence of such evidence, the responsibility for misdeed falls to the corporation's most senior executives -- for failing to create and maintain a process that should have captured and revealed such misdeeds, thereby PRODUCING THE MISSING AUDIT TRAIL.
This pass the buck attitude really pisses me off. You're the head of the company, department, whatever, it's your gawd damn responsibility if your subordinates do something wrong. If you hired bad people it's still your responsibility. If you hired people that hired people it's still your responsibility. I don't care if you knew or not. You set the tone, you set the impossible deadlines or requirements.
You fostered a culture where someone thought that would be okay to do. Own up!
He was very old or simply lying. Very first thing University of Washington Center for Information Assurance and Cybersecurity (accredited by U.S. Department of Homeland Security, whatever that means) teaches you about becoming a CIO is precisely delegating responsibility :)
If you think this hasn't been endemic in this industry, let me take you back to 1998:
'Car manufacturers can use modem electronic equipment to adapt the engine to any type of test cycle. They can even tell the computer of the car how to recognise when the car is being driven according to a specific test-cycle and adjust the combustion accordingly. It was this kind of software that six truck manufacturers, including two European firms (Volvo and Renault), recently used in the United States to defeat the EPA's pollution control. As a result emissions of nitrogen oxides from highway driving increased by 300 per cent. " extract from: 'Cycle Beating and the EU test cycle for cars' - Per Kageson - European Federation for Transport and the Environment - November 1998
Which further begs the question: if I could dig this up with literally 60 seconds of Googling form vaguely recalling having heard something like this before, how come this hasn't been brought up by any of the 'journalists' on the case?
I just today listened to the BBC Bottom Line Special [1] on this, and their guests, one who used to run Jaguar, and an expert on VW, don't buy it. Their argument is that the engineers in VW could have done a proper job, but the technology would add another $500 to each vehicle, and VW must have been looking for a way not to spend the $500. And all the other VW engineers would ask "How did you do it?".
They actually did have an answer to "How did you do it?"
The idea was that only DEP and EGR was needed for the smaller TDIs simply because they burned less fuel so produced less emissions across the board. That's not enough to meet current limits, but VW claimed they had better CDI that solved it. They got many patents related to that and other companies even licensed it. But how did they get such good performance, the lines was due to the turbo and the tests did not really spool that all that much.
Turns-out it was baloney, but that's how they explained it at least.
"Now, I'm not telling you to do anything illegal, but if you don't pass those emissions standards on the next deadline, you're fired."
At each level down the chain.
I actually saw this in action at a big company subcontracting division to two younger employees (one male and one female): "We need that contract. Period. Your expense account on this unlimited and unaudited. Make sure you go pull enough cash that you don't have to put anything embarrassing to the company on the card. I'm not telling you to do anything illegal, but if you don't close that contract, you don't have a job Monday. Anyhow, make sure the client has a good time this weekend. See you Monday. Maybe."
They closed the contract. No one asked what it took. I left that job post haste.
"This was a couple of software engineers who put this in for whatever reason"
This was a system. A complex one. No way it was done by a couple of guys at 2AM, or whatever. This has to have been inter-group collusion, and management is in CYA mode and shoving individuals under the bus.
VW has gone from being a cravenly stupid company that is going to be fined billions of dollars, to a place where I suggest it is in every engineer's best interest to resign, and the earliest opportunity they can manage.
Say the detection for the emissions testing is something like a flag in the software that says the car is on a dyno, or some other factors that the ECU has legitimate reasons to keep track of.
In that case, the defeat system is just a quick if else predicated on those state variables and swapping between a set of engine parameters that is emissions test safe and another set that provides better performance.
I don't mean to insist that this must be the case, I'm just trying to illustrate that the cheat could happen at a level where much of the complexity has already been abstracted away.
Okay, imagine you're working on the hardware of the engine. Pistons, bolts, lubrication. One Friday afternoon you leave work depressed because the emissions suck, they've sucked for months, and they won't get better without a major redesign.
On Monday morning, the emissions tests are suddenly passing.
"Ha, just one of those things," you say, "I thought we had emissions problems, but I guess we don't after all. Let's ship the engine and have a party!"
I'm not arguing that it was a few people working in secret, just that it would be possible for the changes to be done at a high level (in the engine system), without pulling in all the people that were involved in developing the vehicle (or even the powertrain).
"On tonight's episode of Punked: watch as we add a second floor and backyard porch to this man's bungalow while he's away for the weekend! You won't believe his reaction!"
What a classic terrible management move. I worked at a ski shop in college that also sold (really bad) used ski equipment on eBay. One time the lead manager accidentally let a lot of stock go live with a $1 minimum. They realized it and went back in under different accounts and bid them all up. When eBay caught them, they blamed the employees for it (myself and 4 others), who all had nothing to do with it.
We all quit the next day. Who wants to work for someone like that?
Do we think VW are the only car manufacturer doing this? I very much doubt it. They're just the first to get caught.
Where rules stand in the way of profit, they get bent or broken. Banking, automotive industry... which next? Oil? Housing? Pfff ... come on, we all know they're rotten, it's just a question of how rotten and whether it comes out.
I suspect it's the tip of the iceberg for the auto industry. Whether we give a damn, and whether we can or will do anything about it remains to be seen.
Why are so many people keen on defending VW with the claim "if VW are doing it, everyone else must be"? That dilutes the severity of this problem. I've seen several threads on HN meant to do this.
>Why are so many people keen on defending VW with the claim "if VW are doing it, everyone else must be"?
Well for one reason, sanctioning one manufacturer for behavior that all manufacturers engage in means that selective enforcement can be used to pick off unwanted competition in the marketplace.
Not to defend VW but "the severity of the problem"? Let's put this problem in context: VW cars may have more emissions that we used to think but at the same time a large part of the population insists on driving ridiculously oversized trucks which guzzle breathtaking amounts of gas for no good reason. I think the outrage is completely out of proportion.
Buy one directly from Germany, in contrast to what the US division CEO is saying, people are pretty much taking the fall in a top down pattern at the headquarters. (golden parachutes do play a role in this though)
Expect VW US to play dumb until something out of Germany is officially done. VW wont go out of business, but they are 10 years from growth in the US and probably most other markets. I think the real fallout will come if the prestige brands are made to look bad. Audi and Porsche have much more to lose, and those are the profit centers for VW.
I heard (total hearsay) from a friend in Detroit that GM spent millions of dollars over several years trying to replicate VW's diesel technology. They couldn't make the numbers work. Now we know why.
By the way Mr Horn is head of VW Group of America, a subsidiary which besides operating a factory in the U.S. is more or less a distributor of cars for VW and related brands in the U.S. As other people have said Horn doesn't know what's going on at VW AG.
Also, judging from an earlier article it appears it's common to refer to various higher-ups on the product teams as Engineers even though they are quite old and were clearly acting in a managerial or even executive capacity. When Horn says "software engineer" he's making a leap and it's the wrong leap.
Horn is making these statements after being pressed by members of Congress, he's not volunteering these misleading statements.
How could some engineers cheat the entire VW corporation?
Germans are known for their ridiculously meticulous standards and testing. VW should have many in-house and external-independent labs performing tests.
The engineers who write the ECU code have a great distance from the mechanics who test every engine and car batch from their production line.
How did they never tested and engine entirely; power, emissions, consumption?
You may fool the regulators but you can't fool your own mechanics.
"Some people have made the wrong decisions in order to get away with something"
Some people; he doesn't actually say it was the programmers. Both statements are perfectly consistent with the cheating being orchestrated from the top. This looks like a very careful attempt to mislead without actually risking a visit to Club Fed.
He may come to regret testifying under oath with these particular sentences, which are very much falsifiable. I'm not sure what the penalty is for lying to congress but this could very well backfire. There is absolutely no way that anybody within VW is able to put limits on the scope of the fall-out of this whole affair at this point in time.
Not smart to still be in 'damage control' mode when the only way out is total transparency.
I'll be very surprised if there won't be a mass resignation of engineers working for VW for being thrown under the bus like this.
The mere suggestion that 'a couple of engineers and their managers' could get away with deception on this scale speaks volumes about VWs perceived internal lack of process - something they shouldn't be able to get away with - and their lack of documentation for the development track of the engine.
If they've shredded documentation or if they suddenly find their version control systems and email audit trail blank there just might be jail time for some execs after all.
The thing that bugs me is that they are still somehow in lala land about how they're going to cover this thing up, digging a deeper hole in the process. You'd say the VW brand has sustained enough damage but clearly the execs don't agree with that.
"for whatever reason"... quote of the day, as if engineers put stuff in production without reasons, specifications or instructions.
I think VW gets a bit of an unfair beating. Yes, your car probably looses in value, but
- the cars could be upgraded (which VW seems to be willing to do for free) and
- no one was physically hurt (not like the General Motors ignition that actually killed drivers).
Also when seeing the top 3 bestselling cars in the US being sheer gas guzzlers (Ford F, Chevrolet Silverado and the Dodge Ram), the word 'hypocrisy' seems like a fair description to the public reaction in the US.
That being said:
Companies being social constructs and thus complex in their decision making process, framing this scandal as an orchestrated scheme by VW's complete management is ridiculous. What impresses me is, that this Scandal was in the making for months and they did not do anything in advance. Somehow, to me this proves the statement, that they weren't involved in this.
VW's old leadership was notorious for being hands-on on even slightest car details. Specifically Ferdinand Piech - who was nicknamed "Fugen-Ferdi" because of him insisting that VW car's "Spaltmaße" (clearance?) be the industry's best - is very unlikely to not have known about this.
IMO some did know, and choose to not inform those that didn't know - not even when it was clear to them what this would cause to the company.
In my country Audi says the cars using the engine are "technically safe to use and drive". I did report that to the consumer oversight board. I believe it's false marketing, the cars are a danger to pedestrians and cyclists outside the car.
I believe it's lies upon lies from the upper management seems like a corporate culture going wrong.
There's clear evidence that many cities throughout Europe have too high air pollution, and that it kills close to what smoking does. Cities have been ordered by the EU to clean up by the end of the year, or face harsh fines. Over 80,000 people die in the UK per year (smoking kills 100,000). Mostly from diesel engine pollution.
Now try to think how many people it kills in the rest of Europe, the USA and the world?
Cars (and buses) stink, are noisy, and worse -- they kill on a massive scale. There is alternative tech available for order now that doesn't kill as much, and provides better service.
Demand that the killing machines are replaced, and the makers, and the users are brought to justice if they don't!
That's evidence of pollution in general being a problem, which isn't even in the same ballpark as establishing culpability for any amount of deaths for an extra amount of one kind of pollutant by one carmaker over a fixed time period.
Correct me if I'm wrong, but that article talks about particulate emissions, something which even the cheating VWs don't have a problem with - the datapoint that led to this whole thing coming to light were high NOx emissions, not particulates.
These are worrying times for VW if the new CEO to say something like this. Many VW engineers will be considering leaving if VW is going find a couple of engineers as scapegoats, and not be back them up.
Presumably there are a couple of engineers who deserve some of the blame, but wow, what better way to say "Our culture is poison". Both because they're throwing independent contributors under the bus, and because he's saying the culture is such that the engineers would think it's OK.
To be fair, I think he might not have meant it to come out that way.
EDIT: Looking again at the article, it seems like the US CEO isn't even aware of this: "The German automaker has suspended 10 senior managers, including three top engineers, as part of its internal investigation." So VW doesn't necessarily have a poisonous culture. But they do have a US CEO who doesn't prepare when he testifies to Congress.
Unfortunately, we may never know the truth in the scenario. The way that automotive world works with the multi-tier supplier chain, anyone who is not close to the source, will never know where the real blame is.
Behind the scenes in VW's HQ in Wolfsburg, it must be chaos.
"This was a couple of software engineers who put this in for whatever reason"
In other words, it's the programmers, the "geeks," the "code monkeys" that should take the fall for all of this and not the non-technical, MBA-holding managers to whom they answered and who almost certainly enjoy better compensation and more respect.
The perils of our profession's relatively low prestige are on display here for all to see.
Not that it changes your overall point to VW/VAG is a pretty big on having engineers in senior management. Both their former and current CEO have a serious technical background (metallurgy and Computer engineering respectively). As does their chairman of the board.
Let's assume what he says is true. Then the managers of those engineers are incompetent and their managers also. If you hire people who do this sort of thing, and you don't even notice, then the problem is yours. There's no scapegoating here.
Still assuming this is true: If a few rogue individual engineers can do this, imagine what else could be lurking in that codebase.
I'm assuming the development process for embedded auto software like this would be pretty rigorous, meaning it would be fairly difficult to sneak in code like this.
I also assume that testing of emissions software like this would involve a lot of lab/road testing with the vehicles against the EPA test profile and therefore a lot of people would be involved. Suggesting this is the stealthy work of just a couple of software engineers is very hard to believe.
Can anyone who's worked on embedded software in the automotive environment comment?
The investigations regarding Toyotas(?) unexplained/unintended acceleration suggest otherwise. The code was apparently pretty bad, halfheartedly reviewed and just unmaintainable. While there the software seemed to have (a huge number of) errors put in by mistake, of course most independent reviews should also spot intentional mischief.
I don't claim that VW had comparably crappy code, but "industry practice" likely wouldn't have caught it either way.
I've always wondered how something like this plays out legally: Can the engineers be somehow held accountable for writing something they were forced / asked to by their managers?
This isn't a direct comparison, but after World War II it was decided clearly that soldiers are not protected by saying "my boss told me to do it" when the commit war crimes or other atrocities. This is now called the Nuremberg defense and it holds no water at all.
The same principle applies here, in a general sense - but it's sufficiently different in practice, I wonder how it would be treated too.
Engineers aren't solely responsible, but... Did you software guys not take engineering ethics classes? You have a responsibility to society to consider health and safety of the public, even if it means losing your job.
Most 'software guys' have academic backgrounds in computer science, which typically doesn't include anything like engineering ethics. An outright 'Software Engineering' degree might include something like that, but it's definitely emphasized far less than it would be in something like mechanical or civil engineering.
We "software guys" aren't required to do anything at all aside from know how to make software. I can't even imagine how to change that, the tools to learn to make software are federated widely and universities have nothing like a monopoly on them.
My school had "Social Implications of Computing" but it was more focused on research ethics, reference citing, IP, privacy, and security type issues.
Also, since we are CS and not Professional Engineers, we did not have any ethical ceremony like Order of the Engineer.
Some type of Professional Ethics course is part of the ACM model curriculum for computer science (https://www.acm.org/education/curricula-recommendations), but I think how that's fulfilled varies widely across universities.
This is kind of tricky, as I'm assuming that, unlike say a hard engineer that works on a Power Plant or a Bridge, they're not licensed as a Professional Engineer.
In that case the organization would be responsible to a degree, but so would engineers that sign off on something like this. The coder shouldn't solely be responsible, but if you sniff something dirty in the water imo it's your moral responsibility to raise something, or resign. I hope that if I'm in a similar position (probably won't be because I don't work on anything that will kill a person if poorly engineered) that I would react appropriately.
That said, this guy is probably full of crap and is just throwing whoever he can under the bus to preserve whatever stock value is left.
By that reasoning we can order soldiers to torture if we simply fragment the process enough. One to strap in the left wrist, one to strap in the right wrist, one to switch on the gas to the blowtorch, etc.
Yes, they can. Most engineers (in the US) have to take an ethics course that covers this. Just because your boss says do something, doesn't give you a pass.
My job title is "Software Engineer," but I have a Computer Science degree. I use the title because that's what people call it, but it's not an engineering discipline. I had to take some ethics-related course work as part of my general education requirements, but this mostly involved reading some philosophy that vaguely related to ethics.
Definitely had to take an ethics course when I went to Waterloo (in Ontario, Canada). I believe it's a requirement for a university's program to receive Engineering accreditation. At any rate, all engineers took it.
You don't have to have a software engineering degree or an EIT/PE to be a "software engineer". I know plenty of people who have vaguely related degrees like mathematics or statistics that never had to take such courses. And of course there are many autodidacts in the field who studied god knows what.
Even if that was the case (which I doubt), it's the executives' fault for creating a culture which fosters cheating. It's their fault for not being aware of what's going on in their own company. They probably have a culture of "Don't ask, don't tell".
These execs get paid huge salaries/bonuses for being responsible for stuff. But when things go wrong, the first thing they do is shift the blame back down to their workers.
Related side question. A few months before this leaked out I bought a brand new Golf TDI(diesel) and I'm in California where it sounds like my car is technically now illegal?
I have no idea what I should be doing at this point. I've been significantly overpaying on the note but not sure I should now. Does anyone have an idea what this means for the owners of the vehicles in question?
If what he is saying is true (which is about as likely as me getting pregnant), that would be far, far worst than cheating for a test: he is saying two people can put a chemical storage, undetected, and release it according to a complex plan, without any tester, QA, manager finding about it.
If that blatant lie even makes sense to him, he should not be let near an anything industrial.
It's quite possible that the "company" didn't actually know about it because they built a culture where people know what when to break rules and when not to ask.
It's not all that different of a culture than a government agency that doesn't need to be told to do things that benefit the party of the current regime.
An earlier article said that the two who engineered the cheating were the chief engineer and the guy in charge of engines. So whether they engineered it or not, they (and the CEO who has already been fired) are responsible; that's their job.
OK, Volkswagen AG, release the source control log for the software. Let's see exactly who put in that code, and the change log for it. I'd really like to see the unit tests for the defeat code.
It's a good thing that there are stiff penalties for perjury during congressional hearings. Few examples just during this past year, when people lied under oath, got caught, and... Oh, never you mind.
Even if this is true, and I doubt that it is, it's not really that important. The real question is why people in positions of responsibility didn't prevent it from happening.
See, this is why I can never believe anyone when they try to justify executive level compensation. They always say it's cause the executives are "taking risk" and "responsible for the company." And when things are going well, they're quick to take credit. But when stuff like this goes on, suddenly the CEO still gets his golden parachute, and none of the remaining executives know what's going on.
This ridiculous story is not going to hold and they know it so watch for higher ups trying to divest. I mean Bosch told them it would be illegal to use the way they were intending. I surely hope someone is watching who at the company is selling shares.
In order to pull off this little 2-person hack that Horn alleges, those couple of software engineers would have to have a major say over the entire engine design. I call bullshit. This little hack ONLY makes sense in the absence of other major engine components.
The company culture is clearly absolutely rotten and their execs are liars. Employee morale, especially in engineering, must be through the floor.