The best solution I have found for the problem of rewarding prevention, is having leaders repeatedly tell their people that prevention is important. And when performance review season comes around, folks that do preventative work can ask for quotes from people more senior that were close to the work to speak to its impact.
YMMV, and this may not work in every org. It did work reasonably well in a number of technical orgs I was in.
Once you phrase it that way, I start to think it might be literally impossible. How did Aslan put it? "We're never told what would have happened." It's basically the same problem as predicting the future, just starting from a time in the past, with even less grounding in reality. Impossible, right?
Measurement is important, but it isn't everything. If it's your only hammer, you're going to have... exactly the bad time our society is currently having, I guess.
This seems like a place where viewing things as a Bayesian instead of a frequentist could help. We only have one outcome that actually occurred, but that doesn't necessarily mean we can't reason about likelihoods of alternative scenarios. I'm not saying I think it's worth trying to be as granular as "reduced risk of an outage by 10% in Q3" or something like that, but it seems a bit too extreme to assume that we don't know _anything_ about what might have happened.
If we didn't have any intuition at all about potential risks, how could we be taking actions that we think reduce risk in the first place? We (hopefully!) don't try to numerically quantify how productive an engineer in raw numbers like "number of lines of code modified" or "number of commits merged", but that doesn't mean we treat it as impossible for experienced engineers to be able to judge the performance of their subordinates. I honestly don't see this as that much harder to measure than the type of engineering work that does already get rewarded; the reason it doesn't get measured as much is because it isn't valued as much, not the reverse.
It’s important to supplement metrics with estimates, stories about the real world impact of the work, quotes from others close to the work. The less measurable, the more you need to lean on those other tools. Assuming 75% of reactive work can be measured and 10% of preventative work can with any degree of certainty, you’ll reach for those other tools more often for the latter.
Yeah, obviously you have to try to predict the effects of your disaster prevention measures, and if you can use quantitative methods so much the better, but that's not "measurement". Not even close, and especially not in the sense that the people who want to count lines of code yearn for.
And yes, those people are still out there. Some of them have learned their lesson about LoC in particular, but they haven't lost their craving for simple (often simplistic) metrics.
The difference with "regular" engineering is that working features are in fact quite tangible, even if their exact costs aren't. You put a ticket on the board for a button, or a bridge, and eventually you can push the button or drive on the bridge. Not so for prevented disasters. By the time a disaster becomes tangible, it's usually too late.
Yep, I don't pretend to believe that "software engineering" is actually "engineering" in the traditional sense. I still that security concerns in software engineering lies somewhere in the spectrum between "real engineering" and "impossible to meaningfully reason about".
As for the people who want to find a metric as simple as lines of code, I can't imagine we ever find something that will satisfy them, so I'm not particularly dissuaded by the idea of people looking for that not being happy with the measures that I think would be effective. To me, this is a classic case of not letting the perfect be the enemy of the good, and I think that it would be unfortunate for us as an industry if we only considered ways of measuring risk prevention that are 100% quantitative.
> As for the people who want to find a metric as simple as lines of code, I can't imagine we ever find something that will satisfy them
The problem with these people is that they're "satisfied" with the illusion of certainty, until things blow up. We agree on the fundamentals re measurement, but you have to watch out for that effect. Every metric is a chance for someone to implicitly decide that it's the ultimate truth of reality.
The best solution I have found for the problem of rewarding prevention, is having leaders repeatedly tell their people that prevention is important. And when performance review season comes around, folks that do preventative work can ask for quotes from people more senior that were close to the work to speak to its impact.
YMMV, and this may not work in every org. It did work reasonably well in a number of technical orgs I was in.