by yagizdegirmenci on 10/26/24, 6:13 PM with 81 comments
by shadowsun7 on 10/26/24, 11:28 PM
https://commoncog.com/becoming-data-driven-first-principles/
https://commoncog.com/the-amazon-weekly-business-review/
(It took that long because of a) an NDA, and b) it takes time to put the ideas to practice and understand them, and then teach them to other business operators!)
The ideas presented in this particular essay are really attributed to W. Edwards Deming, Donald Wheeler, and Brian Joiner (who created Minitab; ‘Joiner’s Rule’, the variant of Goodhart’s Law that is cited in the link above is attributed to him)
Most of these ideas were developed in manufacturing, in the post WW2 period. The Amazon-style WBR merely adapts them for the tech industry.
I hope you will enjoy these essays — and better yet, put them to practice. Multiple executives have told me the series of posts have completely changed the way they see and run their businesses.
by ang_cire on 10/26/24, 11:37 PM
Business leaders like to project success and promise growth that there is no evidence they will or can achieve, and then put it on workers to deliver that, and when there's no way to achieve the outcome other than to cheat the numbers, the workers will (and will have to).
At some point businesses stopped treating outperforming the previous year's quarter as over-delivering, and made it an expectation, regardless of what is actually doable.
by thayne on 10/27/24, 5:37 PM
It then discusses ways that the factory might cheat to get higher numbers.
But it doesn't even mention what I suspect the most likely outcome is: they achieve the target by sacrificing something else that isn't measured, such as quality of the product (perhaps by shipping defective widgets that should have been discarded, or working faster which results in more defects, or cutting out parts of the process, etc.), or safety of the workers, or making the workers work longer hours, etc.
by bachmeier on 10/26/24, 9:45 PM
As "Goodhart's law" is used here, in contrast, the focus is on side effects of a policy. The goal in this situation is not to make the target useless, as it is if you're doing central bank policy correctly.
by jjmarr on 10/26/24, 8:54 PM
by godelski on 10/27/24, 6:38 PM
Here's the thing, there's no fixing Goodhart's Law. You just can't measure anything directly, even measuring with a ruler is a proxy for a meter without infinite precision. This gets much harder as the environment changes under you and metrics' utility changes with time.
That said, much of the advice is good: making it hard to hack and giving people flexibility. It's a bit obvious that flexibility is needed if you're interpreting Goodhart's as "every measure is a proxy", "no measure is perfectly aligned", or "every measure can be hacked"
by nrnrjrjrj on 10/26/24, 10:31 PM
I might be wrong but I feel like WBR treats variation (looking at the measure and saying "it has changed") as a trigger point for investigation rather than conclusion.
In that case, lets say you do something silly and measure lines of code committed. Lets also say you told everyone and it will factor into a perforance review and the company is know for stack ranking.
You introduce the LOC measure. All employees watch it like a hawk. While working they add useless blocks of code an so on.
LOC commited goes up and looks significant on XMR.
Option 1: grab champagne, pay exec bonus, congratulate yourself.
Option 2: investigate
Option 2 is better of course. But it is such a mindset shift. Option 2 lets you see if goodhart happened or not. It lets you actually learn.
by tqi on 10/27/24, 4:12 AM
If companies knew how to make it difficult to distort the system/data, don't you think they would have done it already? This feels like telling a person learning a new language that they should try to sound more fluent.
by osigurdson on 10/27/24, 12:10 AM
by lmm on 10/27/24, 8:10 AM
Can't it? Amazon may be an exception, but most of the time running without numbers or quantitative goals seems to work better than having them.
by James_K on 10/27/24, 12:36 PM
by deeviant on 10/27/24, 6:20 PM
by thenobsta on 10/26/24, 10:22 PM
This seems to pop up in a lot of areas and I find myself asking is X thing a thing I really desire or is it something that is a natural side effect of some other processes.
by ssivark on 10/28/24, 5:02 PM
by lamename on 10/26/24, 9:40 PM
by mark-r on 10/27/24, 2:27 PM
by yarg on 10/27/24, 9:46 AM
However, it's still better to recognise a problem, so you can at least look into ways of improving the situation.
by satisfice on 10/27/24, 3:58 AM
This is wrong, and the wrongness of it undermines the whole piece, I think:
- A fourth way people respond is to oppose the choice of target and/or metric; to question its value and lobby to change it.
- A fifth way people respond is to oppose the whole idea of incentives on the basis of metrics (perhaps by citing Goodhart's Law... which is a use of Goodhart's Law).
Goodhart's Law is useful not just because it reminds us that making a metric a target may incentivize behavior that makes THAT metric a poor indicator of a good system, but also because choosing ANY metric as a target changes everyone's relationship with ALL metrics-- it spells the end of inquiry and the beginning of what might be called compliance anxiety.
by skmurphy on 10/26/24, 9:20 PM
"I really appreciated this piece, as designing good metrics is a problem I think about in my day job a lot. My approach to thinking about this is similar in a lot of ways, but my thought process for getting there is different enough that I wanted to throw it out there as food for thought.
One school of thought 9https://www.simplilearn.com/tutorials/itil-tutorial/measurem...) I have trained in is that metrics are useful to people in 4 ways:
1. Direct activities to achieve goals
2. Intervene in trends that are having negative impacts
3. Justify that a particular course of action is warranted
4. Validate that a decision that was made was warranted
The implication is that the goal is not to find the “right” metrics, but to instead find the most useful metrics to support the decisions that are most critical at the moment. After all, once you pick a metric, 1 of 3 things will happen:
1. The metric will improve until it reaches a point where you are not improving it anymore, at which point it provides no more new information.
2. The metric doesn’t improve at all, which means you’ve picked something you aren’t capable of influencing and is therefore useless.
3. The metric gets worse, which means there is feedback that swamps whatever you are doing to improve it.
One other important point to make is that for many people, the purpose of metrics is not to make things better. It is instead to show that they are doing a good job and that to persuade others to do what they want. Metrics that show this are useful, and those that don’t are not. In this case, of course, a metric may indeed be useful “forever” if it serves these ends. The implication is that some level of psychological safety is needed for metric use to be more aligned with supporting the mission and less aligned with making people look good."
by rc_mob on 10/27/24, 1:24 PM
by stoperaticless on 10/27/24, 10:14 AM
- Use not one, but many metrics (article mentioned 600)
- Recognize that some metrics you control directly (input metrics) and others you want to but can’t (output metrics).
- Constantly refine metrics and your causal model between inputs and outputs. (Article mentions weekly 60-90min reviews)
Edit: crucial part, all consumers of these metrics (all leadership) is in this.
by bediger4000 on 10/26/24, 7:26 PM
Is Goodhart's Law as useful as you think?
by stonethrowaway on 10/27/24, 2:41 AM
by aunwick on 10/27/24, 2:30 AM