When a measure becomes a target, it ceases to be a good measure - Goodhart’s Law
I first came across the idea of Goodhart’s Law when I was working at AWS. The story I was told, which I’ve since learned has no proof, is that when the British Empire was occupying India. The local cobra population was getting too big. The British wanted to control the numbers. They offered locals reimbursement for each cobra head the locals brought in.
The Brits, thinking they were smart, assumed this would cause the cobra population to shrink as they encouraged the locals to kill cobras. Some enterprising local Indians then proceeded to breed cobras, knowing they could breed them in large numbers, kill them and make a tonne of money from the stupid British. Ethics aside, what the Brits didn’t consider was the second order impact of their decision.
I’ve missed these impacts several times in my career. Choosing to build microservices for a startup with a development team of 1 (me). Fun initially, 6 months later trying to lockstep releases across 4 different codebases… not so fun. Taking on a project I wasn’t that interested in but knew it would be good for my promotion, not thinking about the second order impact of “oh shit, that’s going to take up basically all of my free time for 3 months”.
It’s a simple thing to challenge. Ask yourself, “and then what?”. If you’re making your decision within a complex system, you still might not catch every possible second order effect. But at least you are questioning it. Some of Barry O’Reily’s work on residuality, particularly around stressors, is relevant here. If you’re thinking about an architecture for a system, think about the possible stressors that could impact it. There are no bad ideas. Throw them all into the pot. What are the things that could happen that will cause stress on the system?
Once you’ve got your list of stressors, you can then apply that to your decision. You might not solve all the problems, you might accept some of the risk. But at least you’ve acknowledged them. You’ve acknowledged that your decision might not perfect, but it’s the best you have right now.
If you layer these second order impacts into a fictional person who goes all in on AI, let’s call them Karl. Karl has spent the last 12 months using AI for everything. He’s basically automated his entire job away. He uses it for coding, for note-taking, for summarising his notes, for sending his emails, managing his inbox and calendar. Karl feels great because he’s got an AI to automate away his entire job. Now he can just play golf and get paid for it. The first order impacts are overwhelmingly positive.
Let’s stress this decision. What if, as many people predict, the per-token cost of using LLM’s skyrockets. AI providers have got to get out of their 32 billion dollar losses somehow. With an X number of times cost. Is it still viable? Karl has offloaded his ability to think to a hallucinogenic bag of words. Maybe the only way he can move forward is to pay the LLM companies. Bad for Karl, great for OpenAI.
An alternative stressor is that his employer realises Karl has just proved AI can automate away his entire job. Great for the employer, they fire Karl and keep his automations. Karl is sad. But at least he has time to play golf now? For the AI company, though, they now need to consider the aforementioned stressor because what if token costs go up X amount because someone needs to find 32 billion dollars somewhere.
Now Karl’s company is wishing they had Karl back.
There are a lot of if’s and but’s in that short narrative. The point doesn’t have to be that you believe something will happen. The point is to go wild and consider all possible outcomes. Maybe Karl decides he is happy with both of those risks and he carries on. Cool. But at least you’ve considered the longer tail of impacts.
Ask yourself, “and then what?”. And start to consider the second order impacts.
James