Car firmware scares me at times. One day I started up my Kia Soul and the transmission decided it wasn't going to shift out of first gear. A reboot later and everything was working fine, and it hasn't happened since.
Still though, sort of horrifying. (Hopefully I hit some sort of safety or fallback mode that can only occur on boot!)
It is with a mixture of fear and amusement that I observe the workings of my car's firmware. (I can hear the interrupts fire in my stereo system when I change the volume through the steering wheel control and the music skips, some buffer wasn't full! I almost have a 100% repro worked out. :) )
Edit:
Having worked in Firmware for some time now, I can confirm that the skill level of many embedded developers is not what I'd call stupendous. Now of course most people are, on average, average, but embedded seems like a special case.
The thing is, embedded systems have exploded in complexity in the last few years. No longer are software projects worked on end to end by just a handful of engineers, rather embedded engineering teams are being forced to learn the lessons about properly scaling up software engineering that developers in other areas learned long ago. A project with 16KB of space for code could be written by 2 or 3 developers sitting next to each other, and it was reasonable to keep the entire program state in one's head.
Now days? You can get Cortex M4 boards that look a damn lot like an actual computer. Sure you don't have much RAM, but the code complexity is way up there. You aren't just talking over an I2C bus to a couple of peripherals anymore!
On top of this, more and more features are being shoved into cars through the use of software. I talked to a developer of wiring harnesses for one of the major auto manufacturers, he described to me exactly how the auto companies see software as "the easy part" of things, which means they get the short shaft in terms of resources (test, dev, time, budget, etc), but are expected to bear most of the load of new feature work. (After all, it is so cheap to do it in software!)
FWIW, this developer said he has gone back to purchasing older model cars, he won't buy any of the cars running his own team's firmware.
I have worked in firmware for almost my entire software career and I agree completely. The main problem is that most embedded software programmers started out as EE's (like me) and never learned how to architect or build complex software systems. The ones who took the time to learn CS & SE produce noticeably better code and with their knowledge of electronics can do amazing things, but the rest... well, I still remember trying to explain to an EE developer during a code review why "#define ZERO 0" was still a Magic Number!
Bringing back horrible memories. I ended up with an EE degree, but I was two or three core classes away from a CompE. The software literacy has been nice throughout my career[1], but I was never able to swing into any of the firmware development groups at my previous employer. Closest I got was some systems engineering that covered custom FPGAs and off-the-shelf signal processing hardware. Now I will probably never make it past the gatekeepers for any kind of embedded job.
What the embedded industry needs is more systems engineering. Not taking an EE and saying, "Poof, you're a systems engineer now!", but actual, interdisciplinary engineers who understand systems development concepts, requirements analysis, safety analysis, and everything else that goes along with it. Its primarily an aerospace thing right now, but it really should expand to the broader field.
[1] In fact, at the moment I am a pure software engineer.
I had a Mercedes SLK 350 about ~5 years ago (leased). I was at a stop light in the Chicago loop, and while holding the brake, the engine revved up 2-3K RPMs with no accelerator pedal input. After taking it in for work, I was told it was a software issue and had been resolved.
Hello, I signed up specifically to argue with you (be proud).
I think that may have been an overzealous algorithm giving the car a blip of throttle to stop it from stalling when it is idling.
An engine still needs fuel and air to run when even when there is no load on it and this is controlled by a computer. This computer targets some rpm for idling. If the rpm drops substantially low, due to for example a 'mechanical' glitch like dodgy fuel, then the computer may over compensate with too much throttle, causing the revving you experienced.
I'll double on this. I had an SLK230 ('03) and noticed that when at a stop light it would sometimes rev up to about 2k rpms and come back down. I started watching for it and noticed that it happened right after the rpm needle dropped into the 500 range so I think it was an anti-stall feature in the car.
a) Brake pedal is fully pressed b) Vehicle speed is at zero
If these conditions are met, and you're attempting to prevent a stall, you may want to consider pushing the transmission into neutral to prevent "unintended acceleration". Transmission is drive-by-wire, so could be done by issuing a command over CANBUS.
According to the comments on this thread, that scenario can apply to ANY manufacturer. These problems are inherent to the technology being used, not the users of the technology.
The software has become dramatically larger, dramatically more complicated, and dramatically more important. At the same time, the process around developing embedded projects has not even slightly kept up.
Most of the crappy projects I've worked on weren't crappy because the developers didn't know what they were doing. They were crappy because they were old code bases that were recycled over and over and over again through small hardware iterations, and no team working on a single iteration can ever get permission from management to do the necessary maintenance (refactor, redesign, whatever) -- they are just required to get their little port finished fast. Everybody makes the smartest hacks they can on top of the crap they were given, and each step of the way the system gets worse.
Most companies don't start from scratch -- which is almost impossible anyway, given the low quality of documentation from chip manufacturers -- but start from whatever example code base or framework the manufacturer provides. These are invariably bad, and were developed like I mentioned above (hacks on hacks with each hardware iteration)... and then the hacks for your specific project start.
Then the actual development cycle is pretty much dictated to be the 'waterfall' method since it's tough to be 'agile' with hardware. Proper debugging tools, at the level you would get with desktop software development, are either unavailable or cost more than the company is willing to invest. Proper code analysis tools are the same. Continuous deployment and automated testing are practically impossible unless you have tens to hundreds of millions of dollars to spend on infrastructure.
And there's never enough time for any of it.
And there's never enough documentation for any of it.
At this stage, embedded has reached the complexity of enterprise software running on desktops/servers, but without the process and tools needed to make that actually work.
A few years ago my automatic Honda Accord had an issue where instead of kicking into 4th, it would start jumping between 3rd and 4th rapidly, about once a second. I took it back to the dealer and they claimed that the transmission control chip had "fried" and fixed it for about $15 before insurance.
I have no idea what actually went wrong, it's been working swell for 6 years after that now. My next car is going to be manual though, I no longer fully trust that sort of thing.
> transmission control chip had fried
> $15 before insurance
Was this outside of warranty? Because I've paid _hundreds_ for transmission control modules of all sorts: Dodge, FORD, Honda.
The "chips" they would be referencing are likely surface-mount not socket-mount on anything much newer than 1995. (I think my 1993 EEC-IV has a socket mount for the main processor but I haven't had it apart in ages.)
There is no way a brand new TCM is $15 before insurance; and I just don't see what "chip" they would be replacing that costs $15.
Gotta love Honda transmissions though; they've has had their fair share of rather interesting transmission reliability issues. My personal favorite comes from the Acura TL.[0]
"...as the third clutch pack wore, particles blocked off oil passages and prevented the transmission from shifting or holding gears normally. The transmission would slip, fail to shift, or suddenly downshift and make the car come to a screeching halt, even at freeway speeds..."
I honestly don't know, the car was a 2002, this was in 2007. I bought it used a year before (from by father). IIRC it had 115k miles on it when I bought it. I've been able to get another 60k on it since then.
I've never quite bought the "chip fried" line. My suspicion at the time was that this was a frequent issue that they were trying to keep somewhat under wraps (hence the near-free repair), but now I am wondering if there was a firmware issue that they fixed by reflashing something.
That or they couldn't find the issue and it resolved itself when they reset something...
I wish I had pressed for more details, but I was mostly just happy the transmission hadn't killed itself.
>he won't buy any of the cars running his own team's firmware.
These cars run in the millions in production numbers and are driven everyday, all day, by various types of drivers in various conditions. Where's the big firmware mess? Ignoring the limited case of this Prius issue, which seems to have a lot to do with floormats being stuck to the accelerator, I'm just not seeing it.
I think there's a problem of visibility here. If you've worked in, say, fast food, you might not want to ever eat it. Software is the same thing. You get to see how the sausage is made. That doesn't mean that the sausage is unsafe. Or that older methods, you didn't witness or were part of, were better.
You drive the rear wheel drive V8 that weighs 2x my car's weight and I'll drive mine with front drive, ABS, traction control, and incredible MPG. There are a lot of reasons to own a classic car, but safety and efficiency aren't those reasons.
I think the biggest problem with software is that it can be perfectly safe and work OK in 99.9999% of all cases, and still contain deadly bugs that strike in very specific and difficult to reproduce circumstances. That the software works very well most of the time only adds to the belief that there must be another explanation for what happened (I think even Woz could reproduce the runaway accelerator bug in his Prius?)
That a piece of software is widespread and works good most of the time simply can't be used to say whether the software is "safe". To say anything about that, you have to look at the process used to develop the software, and see whether that process has a good success rate of predicting and uncovering software bugs.
If Toyotas code is as bad as this article makes it out to be and the certification standards are so poor, I won't be surprised if we see more cases like this in the future, although they will still be rare.
> You drive the rear wheel drive V8 that weighs 2x my car's weight
Actually, a lot of newer cars weigh more, not the other way around. I believe that this is mostly due to additional safety devices. For example a 1967 Mustang weighs 2500-3000 lbs[1], while a 2014 Mustang is 3500-3700 lbs[2]. A 2014 Toyota Prius is 3000 lbs[3], while a 2014 Corolla weights 2800 lbs[4]. A 1980 Corolla weighed 2000 lbs[5]. As you see, that rear wheel drive V8 weighs the same as the Prius. I think the improvements in MPG mostly come from improvements in engine efficiencies, not weight savings. I don't know what kind of car you drive, but I'd be surprised if it only weighs 1500 lbs.
That being said, I mostly agree with your other points.
Well, yes, cars are heavier thanks to life saving crumple zones and other safety engineering. I'd rather crash in a prius than a classic mustang.
That said, its also a little unfair to compare a 4 door sedan to a 2 door sports car. Modern cars are still heavier, but the difference is a bit more sane.
I think my dad's Cadillac when I was growing up was over 5,000 lbs. The 70s and 80s certainly had heavy cars, but heavy and death-traps and super shitty mpg.
My car (Kia Soul) weighs 2700lb and is considered light. Indeed the newer models have been steadily increasing in weight as more and more features are added. Of course part of this is the consumers fault, more features means more parts which means more weight! The 2012 version of the Soul is currently up to nearly 3000lb!
What's this do for your willingness to use a driverless car?
My outsider's impression is similar to your insiders - the complexity of external systems is catching up with embedded systems, but perhaps not the practices and talent.
To be fair, there's bugs in humans too. Chances are (especially over time) the bugs in a computer will be worked out to the point that they are rarer then the ones in humans.
Google has never written software before that causes people to die if it fails. It's a very different thing from other kinds of software that you just reboot if it is acting funny.
That is exactly I would not trust firmware written primarily or entirely by Google. For all the talk of clueless EEs writing code in this thread, I have horror stories of clueless SEs messing with hardware.
Umm, you can A/B test a new logo or colour scheme. You can't A/B test a collision avoidance algorithm. What makes Google a successful ad platform doesn't translate to making them a good safety-critical engineering company.
> embedded systems have exploded in complexity in the last few years
I feel like something has to give soon. I don't think we can sustain the exploding complexity of software systems too much longer. The cognitive load will be too much to bear to develop an "average" software system soon, unless we come up with some really fundamentally different concepts to manage complexity.
Maybe "systems" are the problem. Systems are almost by definition infinitely complex. Maybe there's a better model that reduces the "system" effect? Could it be a functional approach?
I just want to say: computer controlled cars have saved more lives than any other parts in a car. How different would the automotive world be without ABS and similar devices?
ABS is a great example of a simple embedded system. Initial implementations were relatively self contained.
Personally I'd be spun out in a ditch about 5 times a day without traction control, I tried turning traction control off in my car once, and it turns out the way I drive is basically 100% dependent upon traction control functioning!
Still though, sort of horrifying. (Hopefully I hit some sort of safety or fallback mode that can only occur on boot!)
It is with a mixture of fear and amusement that I observe the workings of my car's firmware. (I can hear the interrupts fire in my stereo system when I change the volume through the steering wheel control and the music skips, some buffer wasn't full! I almost have a 100% repro worked out. :) )
Edit:
Having worked in Firmware for some time now, I can confirm that the skill level of many embedded developers is not what I'd call stupendous. Now of course most people are, on average, average, but embedded seems like a special case.
The thing is, embedded systems have exploded in complexity in the last few years. No longer are software projects worked on end to end by just a handful of engineers, rather embedded engineering teams are being forced to learn the lessons about properly scaling up software engineering that developers in other areas learned long ago. A project with 16KB of space for code could be written by 2 or 3 developers sitting next to each other, and it was reasonable to keep the entire program state in one's head.
Now days? You can get Cortex M4 boards that look a damn lot like an actual computer. Sure you don't have much RAM, but the code complexity is way up there. You aren't just talking over an I2C bus to a couple of peripherals anymore!
On top of this, more and more features are being shoved into cars through the use of software. I talked to a developer of wiring harnesses for one of the major auto manufacturers, he described to me exactly how the auto companies see software as "the easy part" of things, which means they get the short shaft in terms of resources (test, dev, time, budget, etc), but are expected to bear most of the load of new feature work. (After all, it is so cheap to do it in software!)
FWIW, this developer said he has gone back to purchasing older model cars, he won't buy any of the cars running his own team's firmware.