Goodhart's Law and Designing Good Systems

submited by

Style Pass

2024-12-22 17:00:22

The adage above is called "Goodhart's Law." Goodhart is an economist, and the law's context was initially based on economic policies. However, we can apply this law to almost any system, big or small.

Goodhart's law describes scenarios where the way of evaluating a system becomes the target itself. Goodhart's law applies in many cases. If a hospital solely tries to reduce waiting times, they might see reduced quality in patient care. If a school prioritizes high grades in standardized tests, it might observe a deficiency in the completeness of students' knowledge and development. If a business only focuses on driving up the number of sales, it might witness significant churn if customer satisfaction is not accounted for. In the Wells Fargo scandal, employees opened millions of fake user accounts to meet sales targets. This occurred between 2002 and 2016. There are other real-life examples where the measure of success for a system solely becomes the optimization objective, thereby causing unintended consequences.

I like to think that most things can be framed as systems, and almost any system starts with an intentional goal. However, by solely optimizing for this goal, the system may begin to experience unplanned surprises.