I agree with everything you say except the bit where the failure rate is a fantastic KPI. It has room for improvement.
> they have a requirement in most areas to be airborne within 30 minutes when called. [...] "percentage of the time where maintenance can't get an aircraft ready to be flying missions in less than 30 minutes".
The problem with this KPI is that it's completely arbitrary. Why does it have to be 30 minutes, and not 25, or 35? Changing this threshold might change your failure percentage significantly. This KPI is not actually a ruler that measures your performance – it's using your performance as a ruler to measure the threshold, because the easiest way to change the KPI number is by changing the threshold, not the performance.
After all, wouldn't 25 be better than 30? The only reason it's 30 is that someone at some point said it has to be 30.
An arbitrary cutoff limit like 30 minutes (what Deming would call a numerical goal without a method) has one of three effects:
- Either it's really hard to accomplish, in which case the large failure rate is demotivating, or
- It's just about what you can do, in which case it's meaningless anyway, or
- It's easier than what you could technically do, which might inspire dilatoriness: "You can't complain; we're meeting our goals!"
Instead of comparing yourself to arbitrary thresholds, use the raw metric as a KPI instead. Use "minutes until airborne" as the KPI. That you can optimise indefinitely, and it is an honest representation of your current process, not how it compares to a number someone threw out at some time.
I know, I know, the 30 minute number is not entirely arbitrary, it's probably related to how easy it is to keep a person alive. But it's arbitrary in the sense of being a cutoff – for keeping people alive, shorter is always better. There's nothing magical happening at exactly the 30 minute mark.
> Instead of comparing yourself to arbitrary thresholds, use the raw metric as a KPI instead. Use "minutes until airborne" as the KPI. That you can optimise indefinitely, and it is an honest representation of your current process, not how it compares to a number someone threw out at some time.
This is an example of a more general phenomenon - when you have outcome buckets, it's generally better to do your modeling on raw outcomes ["it took us 32 minutes"] than to try to do it on the bucketed outcomes ["we didn't make it in 30 minutes"]. Bucketing throws most of your information away.
Andrew Gelman talks about this all the time in the context of election modeling, where it's better to predict how many votes something will get, which is a continuous variable (and then, based on the predicted vote, predict whether the thing will win or not), rather than trying to predict whether the thing will win, which is a discrete variable.
Average minutes to airborne? Median? Mode? 90th percentile? 99th percentile?
Optimizing for each of those would result in a different process, different outcomes, and quite possibly worse outcomes for the stakeholders than "proportion under 30 minutes" which is just a sixth way to slice the same data.
> Average minutes to airborne? Median? Mode? 90th percentile? 99th percentile?
Facetious answer: yes.
More useful answer: you want to record the full timeseries of individual data points. From this you can derive mean, median, mode, 90th percentile, 99th percentile and any other statistic that is shaped like hole in your cost–benefit calculations. Including trends and changing variance.
For reporting purposes, the upper process behaviour limit is probably a good one as "the" numeric value of the KPI, but the true value is in the timeseries.
30 minutes is what the 4-Star General who runs NORAD had decided s/he wants to see as a response time.
It's also just about the fastest you can get a Hercules airborne and still include time for flight planning. It takes almost exactly 22 minutes to start a Hercules by the book, and that leaves 8 minutes for taxi and takeoff.
30 minutes is not arbitrary in the context of search and rescue. When someone has fallen overboard on a Naval vessel in most parts of the world's oceans their survival time is as short as 30 minutes: https://ussartf.org/cold_water_survival.htm
In rare circumstances the water can be much colder than average and this drops below 30 minutes, but for the most part the military operates in waters with survival time no lower than 30 minutes.
On the one hand, choosing an arbitrary but good enough target avoids wasting effort on overoptimization and ensures a focus on the worst case, not on the best case; on the other hand improvements are not elastic: maybe rearranging parts and supplies to make them more accessible is enough to make 28 minutes as likely as 30 before, but getting to 26 minutes could require expensive tools and 24 minutes would require different, easier to service aircraft.
Yup. Optimisation is always a trade-off problem that needs to be approached intelligently, not mindlessly. The raw metric of minutes to airborne leads naturally to the intelligent points you raise about what each minute is worth.
The arbitrary target implies a cost function that is infinite below 30 minutes, and zero above it – this, if anything, suppresses intelligent discussion and leads to wasted effort.
> Instead of comparing yourself to arbitrary thresholds, use the raw metric as a KPI instead. Use "minutes until airborne" as the KPI. That you can optimise indefinitely, and it is an honest representation of your current process, not how it compares to a number someone threw out at some time.
You are completely right, but this is much harder to explain to a regular person who doesn't have a math/stats background.
"Get the aircraft in the air in 30 minutes" is much easier to understand, even if the underlying assumption is that the number can be anything – especially when, as you pointed out, there are some reasonable biology-based reasons for 30 minutes.
"Get the aircraft in the air as fast as possible, but then we'll try to optimize that number" is inviting confusion ("Wait, optimize? But we're already working as fast as we can!).
Of course, the tradeoff in the hard threshold is that people might stop improving after hitting the 30 minute threshold. In my opinion this is at least partially based on cultural factors – you should be working in search and rescue at least partly because you want to help people, and should be willing to optimize further to accomplish that goal.
> they have a requirement in most areas to be airborne within 30 minutes when called. [...] "percentage of the time where maintenance can't get an aircraft ready to be flying missions in less than 30 minutes".
The problem with this KPI is that it's completely arbitrary. Why does it have to be 30 minutes, and not 25, or 35? Changing this threshold might change your failure percentage significantly. This KPI is not actually a ruler that measures your performance – it's using your performance as a ruler to measure the threshold, because the easiest way to change the KPI number is by changing the threshold, not the performance.
After all, wouldn't 25 be better than 30? The only reason it's 30 is that someone at some point said it has to be 30.
An arbitrary cutoff limit like 30 minutes (what Deming would call a numerical goal without a method) has one of three effects:
- Either it's really hard to accomplish, in which case the large failure rate is demotivating, or
- It's just about what you can do, in which case it's meaningless anyway, or
- It's easier than what you could technically do, which might inspire dilatoriness: "You can't complain; we're meeting our goals!"
Instead of comparing yourself to arbitrary thresholds, use the raw metric as a KPI instead. Use "minutes until airborne" as the KPI. That you can optimise indefinitely, and it is an honest representation of your current process, not how it compares to a number someone threw out at some time.
I know, I know, the 30 minute number is not entirely arbitrary, it's probably related to how easy it is to keep a person alive. But it's arbitrary in the sense of being a cutoff – for keeping people alive, shorter is always better. There's nothing magical happening at exactly the 30 minute mark.