> Goodhart's Law: When a measure becomes a target, it ceases to be a good measure.
This makes me think of the first song I posted to YouTube. I couldn't help but curiously flip through the viewer statistics. Automatically I began wondering how I could prevent the 50% drop-off at X seconds, and what type of person was sticking around beyond the drop-off. And before I looked much into it I realized this would be a great way to suck all the life out of something that's supposed to be fun.
This is one of the many reasons I don't run analytics on my website (in addition to the more obvious ones, such as privacy). In my case, I still get useful "second-order" information from the frequency of people writing emails to me about the content, and it feels a lot nicer to read that someone found something I wrote useful than to try to micro-optimize for what people are willing to click on.
"How to Game the College Rankings -
Northeastern University executed one of the most dramatic turnarounds in higher education. Its recipe for success? A single-minded focus on just one list."
This is a nice list to read through. The concept quoted, Goodhart’s law, came up in a project review today, though I didn’t know it was named. It feels nearly unavoidable, but at least recognition helps.
Keep your eye on the prize, and the prize isn't measured by metrics. Any prize in the real world is something complex and complicated, hard to describe and prone to squishiness.
Metrics are things that are "easy" to measure and compare across systems. Setting goals in terms of metrics is appealing and even necessary because it makes things measurable and comparable in the real world, but you need to have some cross-check where you can come back and verify that you are moving closer to your real world goal.
---
A real world example: Consumer products are often distinguished by appearance, and surface quality is an important facet of appearance. You purchase a surface quality tester and improve the measurements produced by the surface tester to the point where you beat a competitor's surface measurements. Then you have a customer clinic and have real potential customers evaluate the two products again. Customers prefer the competitor product because you were not actually measuring attractiveness or appearance of quality, you were measuring something that the surface tester measures.
If the impact of your work can be losslessly reduced to something that can be directly and cheaply measured, you're probably solving an easy problem or a theoretical one.
Thanks! I had hoped the original article might provide approaches for balancing the positives and negatives of metrics, but didn't find any. The "Building Less Flawed Metrics: Dodging Goodhart and Campbell’s Laws" paper you linked is exactly what I was looking for. :)
We changed the submitted title to that of the page. The submitted title was Hacker Laws: “When a measure becomes a target, it ceases to be a good measure.”, which of course is Goodhart's Law. Some of the comments in this thread are referencing that one.
It's commendable that the author counteracts the tendency towards cynicism with the occasional rebuttal, although to fully embrace the spirit of kindness, I would maybe skip the Dilbert reference.
Remember: If everyone really were as stupid and/or evil as some of these maxims postulate, it'd be highly unlikely that you just happen to be the exception. Also: management, design, law, marketing, journalism, social sciences, and politics are all disciplines just as challenging as programming or engineering.
And if you can't think of any reason why one of those fields should be difficult, you're just as likely to better at it as that field's practitioners being better at your job.
DRY is dangerous if applied to pedantically; sometimes similar code should be allowed to exist to make overall architecture simpler, or allow similar components to evolve (possibly Divergantly) over time.
The Pragmatic Programmer also warns against overzealous deduplication. Stuff that are coincidentally the same should be left untouched. Only remove duplication of knowledge.
Would anyone care about discussing some 'rule' with barely greater explanatory power than flipping a coin? People aren't going to adjust their thought patterns, in practice, for such an improvement.
This makes me think of the first song I posted to YouTube. I couldn't help but curiously flip through the viewer statistics. Automatically I began wondering how I could prevent the 50% drop-off at X seconds, and what type of person was sticking around beyond the drop-off. And before I looked much into it I realized this would be a great way to suck all the life out of something that's supposed to be fun.