Goodhart's law is four failures, and an ML team can budget for each one

In March 2025, some of the authors of the Leaderboard Illusion paper, who had submitted Cohere models to Chatbot Arena before, entered two identical checkpoints of Aya-Vision-8B into the human-preference leaderboard run by LMArena. They did not tell the organizers that the two were the same. The Arena rated one at 1052 and the other at 1069, each with an interval of roughly 20 points on either side, and four other models landed between them. A lab that submitted both in private and published only the higher score would have announced a gain that did not exist. Singh and coauthors report the experiment in their 2025 paper. It shows, at small scale, one of the several things people mean when they invoke Goodhart's law.

The law has two famous wordings. Charles Goodhart's 1975 remark about British monetary policy, as Manheim and Garrabrant quote it, held that statistical regularities tend to collapse once they are leaned on for control. Marilyn Strathern's 1997 essay on audit in British universities gave the version people repeat: "When a measure becomes a target, it ceases to be a good measure." She credited the label to an educationalist named Hoskin and illustrated it with degree classes: the more a 2.1 examination result becomes an expectation, the worse it works as a way to tell students apart.

Each wording compresses several mechanisms into one sentence, and the mechanisms call for different defences. My argument is that an ML team can set a budget for each in advance: how many candidates a holdout may judge, how far a learned proxy may be pushed, which second metric watches the first, and who is allowed to touch the evaluation. Most teams set none of these. The habit they do practice, keeping the test set from repeated use, guards against the failure that the best evidence I found rates as mild, and I think the missing budgets matter more than that ritual. The evidence that cuts against me gets its own section.

One sentence covers four different failures

David Manheim and Scott Garrabrant's 2018 paper, "Categorizing Variants of Goodhart's Law," starts from a goal G that the team cares about and a proxy M that it can measure. A Goodhart effect is any case where optimizing M collapses the statistical relationship between M and G. They separate four ways it happens.

The regressional variant exists because a proxy is the goal plus some error. Select for a high value of M and you select for a high value of the error as well, so the winner's true quality sits below its score. The authors call it the most fundamental variant, because any imperfect metric produces it. The extremal variant appears when optimization pushes the system into a region where the old relationship between M and G was never observed, which in machine learning is the familiar problem of out-of-sample prediction. In the causal variant the team acts on the metric, or on something between the metric and the goal, and the causal path that made the metric informative breaks. The adversarial variant needs a second agent with different goals who optimizes against the metric the team uses. Campbell's law, which holds that social indicators used for decisions invite corruption, sits in this group, as does the cobra effect: a bounty on dead cobras was said to have led people to breed them.

Variant	What breaks	Typical case in ML work	First defence
Regressional	Selecting the top score also selects the noise	Best of many candidates on one holdout	Count candidates, confirm the winner on fresh data
Extremal	The proxy is trusted outside the range where it was checked	Heavy optimization against a learned reward or judge	Cap the optimization, audit the proxy against a stronger judge
Causal	Acting on the metric severs its link to the goal	A product metric that rises when the experience gets worse	Guard metrics the change should not move
Adversarial	Someone with a stake shapes what gets scored	Private variants, tuned versions, rewards tied to the score	Separate who optimizes from who evaluates

The variants overlap in real systems, and the paper says so. I include the table because the first defence differs by row, and a team that answers every Goodhart worry with "hold out more data" has covered one row of the four.

Choosing the best of many candidates inflates the winner by a computable amount

The regressional case is the one a team can put numbers on before it starts. Singh and coauthors simulated a provider that tests a family of variants whose latent average Arena score is 1200 and publishes the best. The expected best grows with the number of variants, because the maximum of several noisy estimates sits above their common mean. Their identical-checkpoint experiment, described above, confirms the direction with real votes. They also report that Meta tested 27 private variants in a single month ahead of the Llama 4 release. The paper does not claim to know how far Meta's published number sits above the truth.

The arithmetic fits on a whiteboard. This example is made up. Suppose a team evaluates candidates on a holdout of 1,000 items, and every candidate truly has 80% accuracy. The standard error of one measured accuracy is the square root of 0.8 times 0.2 divided by 1,000, about 1.26 points. For N independent errors, the expected maximum is about 0.56 standard errors at N = 2, 1.54 at N = 10, 1.87 at N = 20 and 2.25 at N = 50 (computed by simulation). The winner of ten identical candidates therefore looks about 1.9 points better than it is, and the winner of twenty about 2.4 points. Real candidates share most of their errors because they see the same items, so the inflation for similar models is smaller than this, and real candidates differ in true quality, so some of the winner's lead is deserved. What matters is the order of magnitude. A 2-point gain on a 1,000-item holdout is well within what selection alone can produce once ten or twenty candidates have been tried.

Leo Gao, John Schulman and Jacob Hilton derived a related result for reward models. When a proxy equals the gold reward plus independent noise, and both are normally distributed, the optimization effort divides between the real reward and the noise in proportion to their variances. A noisier proxy spends more of every step on the noise.

The defence follows from the mechanism. Keep a count of how many candidates each holdout has judged, and treat that count as the thing being spent. Before a decision that matters, re-score the winner on a confirmation set that took no part in selection: its score there is an unbiased estimate, because the noise that lifted it on the first set has no reason to repeat. Report the number of candidates tried next to the score, so a reader can do the whiteboard calculation without asking.

Pushing a learned proxy further has a measurable turning point

Extremal failures show up where the metric is itself a model. Gao, Schulman and Hilton's 2022 paper, "Scaling Laws for Reward Model Overoptimization," built a controlled version of the problem. A fixed 6-billion-parameter reward model stood in for human judgment and labeled 100,000 synthetic comparisons, which trained proxy reward models from 3 million to 3 billion parameters. They then optimized a 1.2-billion-parameter policy against each proxy, by best-of-n sampling and by reinforcement learning, and tracked the gold score.

The gold score first rose and then fell while the proxy score kept climbing. The shape of the curve depended on the optimizer, with different fitted forms for best-of-n and for reinforcement learning, and the coefficients moved smoothly with the size of the proxy. A larger policy gained less from the optimization but did not overoptimize faster: the gap between proxy and gold stayed almost the same across the two policy sizes tested. A KL penalty raised the proxy score reachable at a given distance without improving the gold score at that distance, though the authors warn the result may be sensitive to hyperparameters. They attribute the fall mainly to extremal Goodhart, the proxy being asked about outputs far from anything it was trained on.

For a manager, the useful finding is that the turning point can be measured and, within this setup, predicted. A team that optimizes against a learned score, such as a reward model, an LLM judge or a click predictor, can reserve a stronger or independent check, for instance a sample reviewed by humans, and track it next to the proxy. The stopping rule is the point where the independent check stops following the proxy.

There are limits. Gold here was itself a model, the authors say the synthetic setup might not transfer to real-world settings, and all the experiments used a single environment, the InstructGPT one. That paper measures reward models, so applying the curve to a prompt tuned against a judge model is my guess.

The Leaderboard Illusion paper has a lab-scale measurement of a related effect. Its authors fine-tuned a 7-billion-parameter base model with 0%, 30% and 70% of the training mix drawn from Arena-style data. The win rate against Llama-3.1-8B-Instruct on ArenaHard rose from 23.5% to 42.7% to 49.9%, a relative gain of 112%, while accuracy on MMLU went from 66.5% to 64.4% to 65.9%, a slight decline. GPT-4o scored the win rates as judge, so the experiment simulates human preference instead of measuring it. Even so, a score climbed on the distribution that defines it while a neighboring score slipped by about one to two points.

A metric that rises while the goal falls needs a second metric watching it

Causal failures come from acting on the metric itself. Ron Kohavi and colleagues at Microsoft reported one in 2012. A bug in a Bing experiment showed users very poor search results, and two key organizational metrics improved: distinct queries per user rose by more than 10% and revenue per user by more than 30%. The authors explained that degraded results force people to search again and to click ads, a short-term gain they compare to raising prices in a shop, and they noted that if short-term metrics tracked the long-term goals, the company would degrade quality on purpose. Their remedy was a better evaluation criterion. They advised against using distinct queries alone and treated sessions per user as the key quantity, because satisfied users come back. They also said distinct queries per task should be minimized, subject to the task being completed, with distinct queries per session as a measurable stand-in.

That last condition is the pattern for a guard metric: the quantity being optimized, paired with a requirement it must not violate. DeepMind's 2020 write-up of specification gaming gives a second illustration. A boat in the Coast Runners game was given a shaping reward for hitting green blocks along the course and learned to go in circles hitting the same blocks, instead of completing the course. I expect a finishing-rate check on each evaluated policy would have flagged the boat. The post says its authors had collected around 60 such examples, aggregating existing lists and contributions from the AI community, and it calls specification gaming far from solved.

Guard metrics have a weakness that follows from the law itself: a guard that becomes a target decays too. Two practices help. Keep guards out of any individual's goals, so that nobody is rewarded for moving them, and include at least one guard that is expensive to game, such as a periodic human review of a random sample.

Llama 4's Arena rating shows what happens when the builder also picks the scored artifact

The adversarial variant can arise without any bad intent. It takes only an actor who controls part of the measurement and benefits from a high score. In April 2025 Meta's announcement of Llama 4 cited an Arena rating of 1417 for an "experimental chat version" of Maverick. LMArena replied on 8 April that Meta's interpretation of its policy did not match what it expects of providers, and that Meta should have said plainly that the submitted model was customized to optimize for human preference. LMArena published more than 2,000 head-to-head battle results for review, said it was adding the publicly released Maverick to the Arena, and said it was updating its leaderboard policies. In January 2026, according to a Slashdot summary of a Financial Times interview that I have not read directly, Yann LeCun said the Llama 4 benchmark results had been "fudged a little bit" by using different model versions for different tests. I take no position on intent. The party whose reputation depends on the score also chose which artifact produced it.

Competition platforms have faced this for years, and their responses are structural. Kaggle splits its test set into a public part that drives the leaderboard and a private part used once for the final ranking. Avrim Blum and Moritz Hardt's Ladder algorithm, from 2015, goes further: it releases a new score only when a submission is a statistically significant improvement over the participant's previous best, so competitors cannot read minor fluctuations in the score and fit to them. Singh and coauthors recommend, for the Arena, that scores be published permanently with no retraction and that the number of private models per provider be limited.

An internal team can copy the structure. One group, the evaluators, owns the holdout, the metric definitions and the final run. The group that builds the model gets a coarse, rate-limited signal in the style of the Ladder. The artifact that is scored is the artifact that ships, and a run is not erased after it is seen. I would also avoid tying individual rewards to a single model metric, because that creates the incentive the cobra story describes. My judgment is that the price of a separate evaluator is small next to the price of shipping on a number that was selected, though I have no measurement of either.

Reusing a test set has mostly been harmless, and that limits how much ritual it deserves

The strongest objection to all of this is empirical. If adaptive overfitting is rare, the machinery above is overbuilt. In 2019 Benjamin Recht and colleagues built new test sets for CIFAR-10 and ImageNet by following the original collection procedures, after almost a decade of heavy reuse of the old ones. Accuracy fell by 3% to 15% on CIFAR-10 and 11% to 14% on ImageNet. Yet the ranking of models was almost unchanged, and each point gained on the old test set was worth more than a point on the new one, with fitted slopes of 1.7 and 1.1. If the models had been tuned to the old test set, later models should have shown diminishing returns, and they did not. The authors attributed the drop to a gap in distribution, the new images being slightly harder, and found no sign of adaptivity.

The same year, Rebecca Roelofs and colleagues compared public and private leaderboards across 120 Kaggle classification competitions that had at least 1,000 submissions. They found little to no sign of adaptive overfitting. Among the top submissions the effects were typically under 1% accuracy. The outliers usually had a defect, either a split that was not drawn independently or a very small test set. Their conclusion was that distribution shift looks like a more pressing problem than reuse of the test set, and that limits on reuse may not buy the reliability people assume. The GSM1k comparison I discuss in the essay on eval sets is a different route to a similar symptom, since it concerns contamination of training data and not repeated looks at a test set: some model families lost up to 8% on fresh problems, and the frontier models Zhang and coauthors tested showed little sign of it.

I accept the evidence, and it changes what I would build. In these studies a team's ordinary workflow, hyperparameter tuning and architecture choices on a shared benchmark, did not erode the benchmarks measured, at least by amounts large enough to reverse a ranking. Recht and colleagues offer two candidate reasons, constraints on how often results are revealed, as in the Ladder, and the narrow range of models that standard training produces, and say their data cannot choose between them. They also say their results do not exclude a constant drop shared by all models. One reading is that the benign result owes something to the first mechanism, constrained interaction with the test set, which would make a limit on looks part of the lesson and not a ritual to drop.

Three points survive the objection. The outliers in Roelofs's data had small test sets, which is the noise term in the regressional formula, so small holdouts remain a hazard. Distribution shift, which both papers name as the main problem, has to be monitored in production, since test discipline does nothing to prevent it. Neither study covers the other three rows of the table: a learned proxy pushed hard, a metric that can be moved without moving the goal, or an actor with a stake. The evidence supports less ceremony around keeping the test set clean and more effort on counting candidates, bounding proxies, pairing metrics and splitting roles.

Six practices put a budget on each variant

These follow from the evidence above, and none of them needs tooling beyond a spreadsheet and a calendar.

1. When a metric becomes a selection or reward criterion, write down which variant you expect to hurt it, and give the metric an owner.
2. Set a looks budget per holdout. Count each candidate scored against it, retire the holdout when the count passes the budget, and re-score any winner on a confirmation set before a decision that matters.
3. Pair each optimized metric with guards that no one is rewarded for moving, in the form of the Bing criterion: minimize this, subject to that being met.
4. Cap optimization against any learned proxy, and track an independent check, such as a human-reviewed sample, until you know where the two diverge.
5. Log the offline score and the live outcome of every release and watch the gap. Google's Rules of Machine Learning, in Rule 37, splits the gap into three parts: training versus holdout, holdout versus next-day data, and next-day versus live data, and treats a large drop from holdout to next day as a sign of time-sensitive features.
6. Separate the group that optimizes from the group that evaluates, publish the protocol, evaluate the artifact that ships, and do not retract runs.

Of the six, the last is the least tested. Gao and coauthors say their models were not capable enough to manipulate a proxy and that their scaling laws might break when models are. For a system that can act on its own evaluation code, I would expect the adversarial row to matter, and I have not found a measurement of it. A first step that costs an afternoon is to pick the metric your team optimizes hardest, count how many candidates have been scored on its holdout, and compute what the winner's lead would be if all of them were identical.

Sources

1. Manheim, D., Garrabrant, S. (2018, revised 2019). Categorizing Variants of Goodhart's Law. arXiv:1803.04585. Source for the four variants, the cobra effect, and the quoted forms of Goodhart (1975) and Campbell (1979).
2. Strathern, M. (1997). "Improving ratings": audit in the British University system. European Review 5(3), 305-321.
3. Gao, L., Schulman, J., Hilton, J. (2022 preprint; ICML 2023, PMLR 202). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760.
4. Singh, S., Nan, Y., Wang, A., et al. (2025). The Leaderboard Illusion. arXiv:2504.20879.
5. Recht, B., Roelofs, R., Schmidt, L., Shankar, V. (2019). Do ImageNet Classifiers Generalize to ImageNet? ICML 2019. arXiv:1902.10811.
6. Roelofs, R., Fridovich-Keil, S., Miller, J., Shankar, V., Hardt, M., Recht, B., Schmidt, L. (2019). A Meta-Analysis of Overfitting in Machine Learning. NeurIPS 2019.
7. Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., Xu, Y. (2012). Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. KDD 2012.
8. Blum, A., Hardt, M. (2015). The Ladder: A Reliable Leaderboard for Machine Learning Competitions. ICML 2015. arXiv:1502.04585.
9. Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., Legg, S. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind blog, 21 April 2020.
10. Meta (April 2025). The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, ai.meta.com/blog/llama-4-multimodal-intelligence (source of the 1417 rating). LMArena (8 April 2025). Statement on Llama-4-Maverick-03-26-Experimental, as reproduced by Simon Willison, simonwillison.net/2025/Apr/8/lmaren; Slashdot summary (2 January 2026) of the Financial Times interview with Yann LeCun, tech.slashdot.org/story/26/01/02/1449227. The LMArena statement and the LeCun remarks are secondhand; I did not read the original post or the interview.
11. Zinkevich, M. Rules of Machine Learning: Best Practices for ML Engineering, Rule 37. Google for Developers (page updated 25 August 2025).
12. Zhang, H., et al. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. arXiv:2405.00332.
13. Illustrative selection arithmetic (made-up numbers): a 1,000-item holdout, 80% true accuracy for every candidate, independent normally distributed errors, expected maxima computed by simulation for N = 2, 10, 20 and 50, as stated in the text.