FYI I used to date a girl with a Masters in CS who got a job with Boeing. She worked on the OpenGL overlay for the B1 Lancer. She wrote about 3 pages of justification for every 5-6 lines of code she wrote. After 2 years she bounced as the job satisfaction in that type of environment, simply wasn't there. Also OpenGL from ADA is... interesting.
She worked at the Boeing San Diego office, which is quite beautiful.
As in: "for every line of code a functional programmers writes" he'll write 5 blog posts and 10 HN comments on why functional programming is better, how currying, functional composition, monads, etc, work etc...
Considering the importance of not crashing, both literally and figuratively, I can understand the need for that heavy of a software process. Isn't this par for the course in high CMMI [1] [2] organizations? I don't really know much about this side of engineering, but I find it fascinating.
CMMI seems to be mostly a moneymaking scheme directed at business types with no skill. In contrast, Boeing used actually effective engineering disciplines.
>>CMMI seems to be mostly a moneymaking scheme directed at business types with no skill.
I beg to differ.
I started my career working for a CMM level 5 software services company. One of the first shocks I received after I entered the web industry was total lack of discipline here. There was barely any documentation in the web industry- No way to keep a track of a impact of a particular change, no design notes, not even a change log of whom to call in case something goes wrong, in the CMM level 5 firm we would have think of roll back scenarios even before we worked on the code.
In the web industry I saw, the idea of roll back planning didn't even appear to people. Yet the way I saw, a little documentation would have the whole team sane, and avoided more 50% of fire fighting we did on weekends.
Rules are boring, but when something large is at stake it generally is worth the process.
> Rules are boring, but when something large is at stake it generally is worth the process.
It's all about finding the right place on the curve, and not being behind it. Add huge amounts of process to most web projects, and yeah, they might turn out a bit better, but they'll cost too much for what the customer wants to pay.
You're right though in that too many web projects have virtually no process or rules, and that's not good either.
Deployment is not development. You can do a lightweight development process and a heavyweight deployment process. At my company we manage our development with a scrum-ish approach but getting anything into production requires an ITIL change.
I worked in a CMMI 5 organization and it wasn't even close to this level for our developers. However, for our testers, project managers and business analysts it's probably pretty accurate.
Testing flight software is very intense. 50 KSLOC of C will take a few dozen man years to develop.
As a side note on terminology, QA means a very different thing in the aerospace industry. The function of QA in this world is to witness the tests and attest that we did indeed run the test as written. Usually in the form of stamping a printed procedure as it is followed. This mirrors the process used to test the mechanical and electrical parts. The testing of the software is called verification and validation testing not QA.
Companies are made accountable if there are human lives at stake. Most of the time there isn't (if there's a bug in some accountancy software which requires the accountant to restart the program, the accountant's life is not at stake) but wasn't there that software bug in one of the Toyota cars (Prius I believe?), where the brakes didn't work anymore under certain conditions? I remember Toyota had to take all those cars back in and had law suits against them, so they were certainly made accountable for the software they wrote for that car.
Correct, but even software that doesn't cause deaths on failure ends up causing real losses (often invisible).
Consider every time Word crashes and work is lost. There are billions of Word users out there, mostly being paid to use it. How much does that cost over the course of a year?
And the accumulated frustration from unreliable software, causing loss of productivity, resistance to change, stress... How much does that cost?
What's ironic to me is that we're shifting the blame from people who attack computer systems - either directly or indirectly through malware - to otherwise innocent ( although demonstrably lacking in one or more competencies ) developers. The developer community won't suffer for the additional rigor any, but still...
Releasing a virus is a tort at best. Cracking a site is a tort at best, and both are probably criminal behavior. Yes, I have this bias.
I've read enough history to know that there have been times when such interlopers ended up as heads on pikes outside city walls. Seems extreme to be sure, but still....
I am a great admirer of C.A.R. Tony Hoare, but virus writers and system crackers give him a "bootleggers and Baptists" common cause with them.
If you have the kind of skills to write Open GL in Ada for Boeing, then you're not the kind of person who's depending on having rote-learn some latest tech (J2EE, RoR, Node, Go, whatever) and only can get jobs looking for that.
Can I ask what her first name was? I used to work the exact same job. They moved all their operations to Oklahoma City and I helped with the transition.
I worked on the B1-B Lancer as well. We never touched Ada with OpenGL. It was mainly C++ with VAPS (http://www.presagis.com/products_services/products/embedded-...). I'm not certain where they're getting the 99% Ada portion as a lot of it is in C or C++.
I got out of that space though. I do mobile application development now.
Why do people seem to think Ada is painful? It's quite strict, but that's a blessing, not a curse. You end up making anything remotely dodgy but necessary very obvious, and in general well founded ideas are clear and concise. You can also define your program much more precisely and make sure that things never go wrong (You can't accidently treat an int and an enum)
I have a possible explanation on this. A lot of people only know Ada in passing, from classes. For someone fluent in another language typically derived from C, the syntax is very new and frustrating initially. People normally comfortable with coding in their preferred language become newbies again, making a lot of frustrating mistakes. And they hate Ada for it.
I've seen several persons being forced into using Ada for a longer time. Geek types, including two long haired ones. They all hated it initially, but after a while they all come to appreciate and enjoy it. Two of them ended up contributing to GNAT. The cross-over time is roughly around 6 months of full time work with Ada, so more than what you get from a class.
So I guess Ada is different enough to be frustrating at first, and most people don't use it long enough to come over the initial shock and appreciate the good sides of the language. They just hate it, and stay with the "Ada is bondage" meme. Too bad.
For any one interested in large software system design and languages, I recommend looking at the "Rationale for the design of the Ada programming language" book. Even if you don't like the result ;) it's an interesting analysis of the challenge of doing large, safety critical system software IMHO. Just read the latest version, the intial one (though interesting) is now dated and made some wrong bets, like using synchronous tasks as the base for concurrency because there was hope that a formal model enabling proving behavior would develop around it, and that never happened.
The reason that a formal model didn't arrive is because it's impossible; the foundation of concurrency is the ability to change state and return the previous state atomically (even if it is only one bit/byte; xref cmpxchg). This necessarily makes the system non deterministic. So building a base for concurrency from synchronous tasks is impossible unless those synchronous tasks are atomic from start to end; most tasks are more than one op code long.
My experience with Ada was that it didn't actually make it any easier to make my code correct, because mixing up an int and an enum when you shouldn't is a trivial problem that's easy to fix in C (never mind that the majority of enums are actually labels for ints); the errors that were hard to figure out and costly to fix were the conceptual ones that were independent of language. Ada just made everything clumsier, more verbose and less readable for no benefit. I suspect on a larger project it would start significantly detracting from correctness by discouraging refactoring.
From memory, he made those comments after he'd been working for a long time in C/C++. He's since seen that language like Haskell and Ada which enforce much stricter requirements have a huge benefit to developers; the inital extra work to learn the language pays off when you get used to spotting even more bugs before you write them because you've been punched in the head by the error messages in the past. You begin to think about code not just from a "how doI get this done?" standpoint, but a "how do I get this done properly, safely and clearly, and make sure the compiler will catch me WHEN I make mistakes". Fixing bugs at development time is vastly less expensive than in testing or after release, so the extra errots can easily be justified.
As far as I see the Ada Doom project is able to do more in less lines of code despite it being verbose (see https://github.com/AdaDoom3/AdaDoom3/blob/master/FOSDEM%20Pr...). For readability, clearly your argument is subjective. Its like saying English doesn't make any sense because it is nothing like Chinese which everyone knows
The sharpest edge of that language meeting bad design and sloppy or inadequate code review. Bad design is a problem in any language, safety critical projects need and typically have much better code review that would have caught the likes of Heartbleed, and every language has its particular sharp edges, e.g. the first launch of Ariane 5 would have successfully reached orbit had the code been written in C. (Yes, it would also have successfully reached orbit had that part of the code been adequately checked - see my above point about code reviews.)
Or more realistically, your experience is on small projects, with only you coding so it is worthless, try opening a large C codebase and fixing all its flaws: good luck.
Ada and its better strict typing is a pain for small projects which is why students don't like it but it really shines on large project (no, it doesn't fix all the issue but it helps).
IMHO what should be done is to take Ada's semantic/strictness and wrap it around a less verbose syntax (1) because if the beginners hate the syntax then the language, no matter how good it is will NEVER take off (and even if it starts taking off it will fall down quickly: see Lisp).
1: yes, I know this was done on purpose but the lack of adoption of Ada EVEN AFTER there was free compilers available show that (IMHO) it was a mistake.
C safety is a oxymoron, which is only enforcable with externals tools.
I also don't regret learning C. It provided me a path to jump into C++ and use its features to close its C underpinnings into a dungeon, as much as possible.
Time for a car analogy. If all languages are cars... I would call Ada's extra checking safety belts - they came with the car from the factory and are guaranteed, within certain amount, to keep you alive. With C you can either make your own safety belts (with duct tape or something), fit on aftermarket safety belts, or you can just be a good driver and not have an accident. I wouldn't call the C car safe, but that's just my opinion : p
This is an awesome article - there are a ton of interesting excerpts all in one place, that I haven't seen before, such as this one:
One compelling reason behind the extensive pre-testing was Boeing's desire to meet the Federal Aviation Agency's (FAA's) Extended Twin Operations (ETOPS) standards ahead of schedule. The original ETOPS rule was drafted in 1953 to protect against the chance of dual, unrelated engine failures. Unless a newly designed and produced aircraft has at least three engines, it usually had to wait, sometimes as long as four years, before the FAA and the Joint Airworthiness Authorities (JAA) will allow it to fly more than one hour from an airport; after a time, the new aircraft is deemed a "veteran" and is allowed to fly three hours away. A shortened trial period would drastically increase Boeing's sales."
It hadn't occurred to me that a 777 is always three hours away from an Airport. I'll have to look more closely on the map.
Thanks - that's a great site. It turns out that 3 Hours can take you pretty much anywhere in the world. This was a nice briefing on ETOPS - http://gc.kls2.com/faq.html#etops-138
PBS did an interesting, long documentary filming the build and certification process for the 777. It's broken into 5 different, hour long parts that can be found on Youtube (https://www.youtube.com/watch?v=0oyWZjdXxlw - First part). The last part goes into detail about what they did to meet FAA regulations in a short amount of time.
Since it's mostly 747 and similarly large planes on intercontinental flights anyway, I doubt this is much of an issue. I wonder if there actually is anywhere in the CONUS that's more than 3 hours from at least a rural airstrip?
The reason why I thought it might be an issue, is that according to Wikipedia, The 777-200LR is the world's longest-range airliner and can fly more than halfway around the globe; it holds the record for the longest distance flown non-stop by a commercial aircraft. . It hadn't occurred to me that because the 777 is a twin engine, it can't really use all that range. It needs to stay near airports.
Twin engine planes absolutely can cross oceans -- if you need three hours' flying distance in the worst case (ETOPS-180), then you can cross up to six hours of water between any two backup airports. North Atlantic routes have only a few hours of contiguous water between, say, Greenland and Iceland. Even the Pacific has enough small islands (e.g. Hawaii) to make something like this possible (shaded areas are outside 3-hour range):
I have developed many years in Ada. It is a marvelous language. Many mistakes are found during compilation. Very often, when it compiles, it works. Since Ada95, the multitasking features are really awesome. The overhead of the verbosity is real for short programs, but becomes less relevant for big projects.
Ada is not in the hype anymore. It is getting more and more difficult (and expensive) to have developers with a good knowledge in Ada.
In my current society, some projects are still in Ada, but the quality of the code is becoming lower and lower, with a lot of type cast. It seems people are thinking in C (C++) and are painfully writing their ideas in Ada.
It is an absolute shame that Ada did not take its place as a dominant player early on. But people have difficulties reasoning about cost in computing. The verification cost for the toolchains pretty much kept places I worked at out of it. It is too bad that those costs were not treated as public goods and paid for out of tax money. I expect it would have been a good investment.
The actual language has little to do with the current widespread use of Ada in critical aviation software. The main reason lies in the certification process and reuse of certified tools. I wrote a little about it here: http://eduardo.intermeta.com.br/posts/2013/2/10/making-aviat...
Dynamic memory management is inadvisable in avionics code, making languages that rely on garbage collection (perhaps pointlessly) challenging to use. But in general, there are no restrictions on what language is used. The particular version of the compiler you are using has to be certified, but just long as you can use the language and the compiler to meet DO-178B/C standards, you can use whatever you want. You can use whatever tools you want, if they get certified. You can write your own tools and certify them. Anything goes, just so long as it meets standards.
(Working on some flight management software, I found a bug in our version of GCC. It's free software, right? So I could just fix it? Or even upgrade to a newer version? That would have meant re-certifying the compiler, so it was easier to work around the bug.)
In practice, I do see a lot avionics code written in Ada. I see a lot written in C. I see a lot written in C++. I haven't seen much else. I would guess a roughly even split between Ada and C or C++, with either C or C++ being increasingly favored for new clean sheet projects.
Surprisingly you don't even need to use DO-178, you just need to convince the FAA that it is safe. I have never heard of anyone not using DO-178 though.
I have never heard of anyone not using DO-178 though.
Me neither. I can imagine that would be harder than just using DO-178, as the auditors are used to dealing with it.
Have you seen level A software written using C++?
Not personally. I have heard of a level A flight controls project that was planning to use C++, and was in the process of getting their C++ compiler (GCC C++, I believe) certified. I don't know how it turned out.
Having had a lot of my undergrad CS classes in Ada (in Toulouse France, the home of Airbus) I get nostalgic every time I see a post mentioning Ada on HN.
I was involved in writing the short term collision alert system for the New En Route Centre down in Swanwick, UK. Basically proving something won't kill is extremely important in this environment. The language is almost irrelevant. Proving every single piece of logic has been exercised (an OR statement had to have true true, false true, true false and false false tests...now add multiple ORs to that). Mission critical software is another world.
First real success was Meteor line 14
driverless metro in Paris: Over 110 000 lines of B models were written, generating 86 000 lines of
Ada. No bugs were detected after the proofs, neither at the functional validation, at the integration
validation, at on-site test, nor since the metro lines operate (October 1998). The safety-critical
software is still in version 1.0 in year 2007, without any bug detected so far.
Pure languages are also great for this kind of thing, because of the lack of side-effects (aka Haskell's moto). OCaml manages its imperative parts in a way that isn't as good had Haskell does, but it's probably possible to get a very clean code that is mostly pure, allowing a mathematical proof of safety.
It'll give an execution count on each statement, and logical expression operands count as separate statements. I've found that when this is used in conjunction with a test suite to ensure that the tests cause every code path to be executed, the code becomes remarkably free of bugs.
I did a little bit of Ada programming, and one of the distinct features it has is you can't do anything clever without being extremely explicit about it. This language has the most aggressive type-correctness checking I have ever seen. When you finally get your code to compile, it's likely that it will work reliably.
Ada uses its type system plus managed memory pools for manual memory management, it's very cool. Safe manual memory management with memory control of C/C++ but the safety of an ML.
People usually think of Haskell when strict type systems come up.. Ada is a "different kind" of strict though.
It doesn't do any of the fancy/confusing inference stuff that Haskell does in order to verify that type conversions/transformations progress along the intended paths, but instead requires you to be very explicit about what you're doing.
E.g. (1) Trying to read something from stdin is a minor challenge, because of how Strings are usually fixed-length and don't interoperate with special arbitrary-length strings. (2) If you want to do dynamic memory stuff yourself, you need to declare your packages as such (with the intent of this showing up during reviews and thus leading to the code being more thoroughly scrutinized.) (3) It allows for creating special sub-types of primitives, like integers of a limited/well-speicified range, so that they can be put into APIs intead of ordinary machine/compiler-dependent integers. (4) Conditions (in IFs and such) don't (by default) short-circuit evaluate, and you have to use speical key words if you want short-circuit evaluation. And the list goes on..
All in all, Ada is quite enjoyable imho. It allows you to write code without thinking too much about it, since you won't be missing (many) corner cases.
On the other hand of course all of this results in more work, which can feel tedious to some - but at least it's not the mind-boggling "what is this *?!" strictness that has you pulling your hairs out in confusion/desperation, that usually accompanies Haskell.
You can make an integer range like 2..15 if you know thats your possible input values, and nothing else, only that will be accepted - no need to parse it yourself or check for upper/lower-bounds/unsigned-ness, the compiler and later runtime does it for you.
Safety and Reliability are not as relevant for most projects as speed, flexibility, familiarity, availability of frameworks and developers, inexpensive compilers/debuggers,etc... All of these are area in which Ada is not as highly qualified as some of its competitors.
But, for Avionics and Space System, Ada is a well known, and continues to be a well known implementation language.
because choosing a language for a commercial project is rarely about safety or effectiveness - it's about language popularity, availability of libraries, and replaceability of the programmers.
The safety and reliability comes at the price of more verbosity and less dynamism. Most programmers don't need (or don't think they need) that level of reliability, and choose not to use languages like Ada (if they are even aware of its existence).
On the other hand, failure is much less of an option when it comes to things like avionics, so much more focus is placed on reliability. In these cases that price starts looking like a bargain, if the use of Ada isn't already mandated (which it probably is).
I was just asking that recently too [1]. I think it's because most "normal" projects, aka web/desktop/mobile apps aren't safety critical, a large cadre of developers prefers dynamic languages that let them program in a completely bottom-up exploratory manner without thinking about types and the like.
Ada is by far the strongest typed language I've used in my 2 decades of programming. In the defense / aviation industry, it's very common to have multiple sub-contractors working on different pieces of the platform. Due to this trait, interfaces between these sub-systems / components are heavily scrutinized. Using a language such as Ada to ensure that large amounts of bitwise data structures are clearly defined and match eases the burden (at least a little).
I recently applied for a job at a company that makes civilian drones (rescue, police). They required knowledge of the Oberon language, I'm pretty sure they use HelyOS [1] for real-time control of their coaxial helicopters.
It's quite interesting how these Pascal derivatives seem to find their niche in high-reliability or real-time applications, although I don't quite understand why. That may have been part of the reason why I didn't get the job.
According to comments there, some modern aircraft run an RTOS (real time operating system) on top of the bare hardware, which then lets you run Ada or C code. Those mentioned include VxWorks[1] for the Boeing 787 and Integrity[2] used by the Airbus A380.
...and now I will always want to fly on an Airbus instead.
Reminds me of the joke:
> At a recent real-time Java conference, the participants were given an awkward question to answer:
> "If you had just boarded an airliner and discovered that your team of programmers had been responsible for the flight control software, how many of you would disembark immediately?"
> Among the forest of raised hands only one man sat motionless. When asked what he would do, he replied that he would be quite content to stay aboard. With his team's software, he said, the plane was unlikely to even taxi as far as the runway, let alone take off.
>...and now I will always want to fly on an Airbus instead.
Reminds me of a joke I was once told by a QA consultant:
His colleague was on his way to deliver a revision of software which had been
extensively tested and many serious bugs discovered and fixed. The software was so
important that it was to be delivered in person by the consultant on a laptop to
the customer.
On the walkway to the plane, he happened to glance up and see the aeroplane he was
about to board was an Airbus. In fact the very revision that had the same
software he'd just been testing and had found to be full of bugs.
Oh no.
Suddenly getting very cold feet and desperately thinking of a way not to get on the
plane he takes a drastic step and deliberately drops and destroys the laptop with
the software he was going to deliver.
Happy now he returns to the office and declares
'well, that's a shame - but I think I'll take the train next time'
- was told to me as a true story, and you never know...
>According to Brian Pflug, engineering avionics software manager at Boeing's Commercial Airplane Group, most companies disliked the idea of a standard language at all, and then seriously objected to Ada as too immature.
By what measure did they consider Ada "too immature" I wonder? It's one of the oldest languages, an open international standard, and has been through several major updates over three decades. On the surface there are few if any languages that claim more maturity than that.
There's no compiler (and standard library implementation) that has faced public scrutiny like other popular languages have. Aditionally, there are no IDEs for Ada comparable to the IDEs of other comercial languages (e.g. C++, C# or Java). Which is understandable given how little interest there is in Ada, amongst the "general public" of developers, which is understandable given how there's no proper free implementation of for Ada.
There's essentially just the AdaCore implementation that costs $$$, the GPL'd AdaCore implementation that requires you to publish your source code (because of the libraries) and then there's some unsupported GCC implementation.
I like Ada, and I would like to see it gain more market share. If it had, things like Rust and Go may have been superfluous. But the way things are, I just don't see it happening. Unless you're some big enterprise Ada is essentially a non-option, due to licensing, and if you're a big enterprise it's essentially a non-option since there's very few devs around for it. And so it remains an obscure language used in very niche applications (albeit with great success there?).
Really strange about the tooling, if I were BAE or Boeing I'd be standing up a project to create an IDE on top of Eclipse - ~2 man year effort for vast productivity gains!
dons asbestos pants in expectation of anti eclipse comments!
What new languages are being developed that could be even safer than Ada for safety critical systems? Are Agda, Haskell, or Rust as safe or strongly typed as Ada?
Weirdly, avionics development seems to be moving away from Ada toward C! I personally think something like OCaml would be a better choice...
I think the problem is, there is so much momentum built up around Ada and C usage in the avionics world that there just isn't the infrastructure and culture built up around better languages, so it's hard to make the switch. Even if you have to write a brand new codebase for some small avionics project, how do you justify the expense of doing it all in OCaml when there is already certified compilers and support for Ada?
Right, that was unclear. Languages that rely on garbage collection can't realistically be used in flight. "Something like OCaml" but with manual memory management might be better than Ada. But with recursive functions also frowned upon in avionics, a lot of the fun of OCaml might be drained...
Probably the coffee machine. Although given the effort required to write safety critical code, I wouldn't be surprised if the coffee machine had more lines of code than the rest of the airplane.
The galley is a main source of in flight fires. I'd expect such systems to be classified as critical to flight safety. Obviously not as much as the auto pilot, but it is still a system that could bring the plane down if an oven or cofee pot shorts. IFE, on the other hand, is not.
I wouldn't be surprised if the coffee machine didn't have a single microcontroller in it, but a simple old-fashioned analogue controller. The prevailing attitude in safety-critical systems is "if it works (and has been certified with much $$$), leave it alone".
The hardware access routines, real time operating system, and bottom level I/O can't be written in ADA. CPUs accept only machine code, so those programs have to be written at least partly in machine code.
Likewise the compiler has to generate CPU instructions,the opcodes and operands of which are embedded in the code generator as binary data. That is code that isn't in ADA either.
Ada get compiled to machine code, so of course it can run directly on the CPU.
There's also no reason it can't be used to write a compiler and there's probably several Ada compilers written in Ada. I'm not sure what you mean about binary data having to be embedded in the code generator; you can easily generate arbitrary binary data from almost any language.
> The hardware access routines, real time operating system, and bottom level I/O can't be written in ADA.
I disagree, Ada has fantastic tools for writing these sorts of code; you can precisely define data types down to the bit level and their precise locations and endianess in memory. Have a look at pages 50-68 of [1] to see these in action. you don't have to know Ada to be able to see what's being defined, and that's it exactly matches the specification of the memory mapped IO registers. Some reference c code is given, but there is no guarantee that the compiler will do what you want; the Ada language does guarantee it.
She worked at the Boeing San Diego office, which is quite beautiful.