I take great care to stay vigilant for these opportunities. Recently, I found myself in a sadness for the lack of changes I was capable of making to our life’s systemic dysfunctions - be it natural or artificial - and found my time was much better spent improving the lives around me directly, individually or in small groups.
I’ve long had the philosophy that the world has enough problems and that it’s not my place to add to them, but this philosophy also gave me a motivation to move mountains for solutions - and would be upset when I inevitably couldn’t. I think this perspective has been the best middle ground between what I’m capable of, and what I want to accomplish.
In my +- 25 years of experience, I've encountered exactly 1 compiler bug. That cost me a full week of debug. Unfortunately, it happened during my very first year as a developer, and that triggered sub-optimal debugging practices (ie: is that the compiler fault again?)
At the end of the day, you want to optimise for debug time. There's the probability that it's a compiler/developer bug (respectively very low/very high) and the time it takes to rule it out. It's of course best NOT to start by investigating the compiler bug.
I find several every year. Most people simply don't see compiler bugs because they don't test (and most of the time don't need to) their compilers extensively enough (by testing I mean they must build, run the generated binaries, and compare the output ).
They most likely do (just look at the gcc bugzilla).
I find them because of a combination of :
- a very large codebase
- compiling with optimizations (O3) and targeting recent archs
- yearly compiler upgrades
- an extremely extensive testsuite
I have found all kinds of bugs (frontend, middle, backend) - the codegen ones tend to be very nasty to diagnose.
> Otherwise it would imply large companies (Google-scale) experience thousands of compiler bugs per year...
This seems likely to be true, from my experience.
Fortunately, most compiler bugs I run into (mostly with gcc and llvm) are not code generation bugs (which can eat months of debugging), but just segfaults / rejecting correct code / other broken stuff.
I also suspect most programmers spend most of their time staying more or less on the “straight and narrow”. There’s lots of obscure features in most languages (especially in old languages like C++) that not many programmers use.
These features are much more likely to have buggy edge cases in the compiler. And only a small select group of programmers will run into those bugs over and over again.
Eg complex template metaprogramming - which had known broken corner cases in all C++ compilers up until a decade or so ago.
Fun. Basically, it decides that (due to UB) it can simply not materialize the argument and carry on.
On Darwin/x86_64, this actually means that it prints out bits from an "uninitialized" register, specifically %rsi in this case, since that's where the second argument should be (even for variadic functions).
Certainly a jarring failure mode! However, if updating your compiler causes you to run into this, you already have a bug in your code (one that clang -- arguably not forcefully enough -- already warns you of).
Yeah I was debugging an issue that was occurring in a Kubernetes environment last night... kept wanting to blame the out of date Istio version. I just could not trust it and that lack of trust stopped me from seeing the whitespace error that was in front of me the whole time...
I think it's really dependent of the domain you are working in. Some stuff is better tested than others. In my 5 years careers I found bugs (found and confirmed by compiler authors) in gfortran, xlf and pgfortran. So yes bugs in compilers are rare but sometimes you just keep running into them.
When facing such large scale issues, communicating properly is very hard: Several teams might be investigating several possible root causes in parallel, and you might change your mind over time as to what is the most probable root cause.
So you might end up communicating something ("we think it comes from X, we're fixing it that way"), just to find yourself changing you mind a few minutes later.
Changing your message is usually not well perceived, even though that's actually normal during an investigation.
I would not like to be in charge of the communication. Finding the balance between saying too much or too little is tricky.
Maybe the right question is: how do you become a great boss?
Getting promoted to an EM position is one thing. Getting good at it is... very hard, and requires new skills. There's very little in your day to day job as an engineer that prepares you to become a good EM.
