The eval set is the specification

In May 2024 a team at Scale AI released GSM1k, a set of new grade-school math problems written to match the style and difficulty of GSM8K, the benchmark that most labs had been quoting for elementary reasoning. The authors matched the two sets on solve rates among human annotators, number of solution steps and size of the answers. Then they ran the models. Several families of open and closed models scored up to 8% lower on the fresh problems than on GSM8K, and the models that were more likely to reproduce GSM8K items from memory showed bigger gaps. The frontier models showed little of this, which is worth saying plainly, because the finding is more specific than "benchmarks are broken." A score is a measurement of performance on a particular set of problems, and when the problems leak into training data the measurement stops describing the skill.

In April 2025 a group led by Shivalika Singh and Sara Hooker examined Chatbot Arena, the most visible human-preference leaderboard, and reported that a few providers had been able to test private variants before release and publish only the best score. They counted 27 private variants tested by Meta ahead of Llama 4. They estimated that Google and OpenAI each received about a fifth of all the Arena data, which matters because training on that distribution raises scores on it.

Neither result means public leaderboards are useless. They are fine for building a shortlist of models. What they cannot do is tell you whether a model is good at your task, because your task was not on the test. For that you need a set of cases you wrote, labeled and defended, and my argument in this essay is that for an LLM application that set takes the place of the specification. It states, in executable form, what the system should do on the inputs you care about. Everything else in the project, including the prompt, the retrieval index and the choice of model, is an implementation that has to satisfy it.

Start from the failures you can see

The method I trust most is also the least glamorous. Hamel Husain, who writes about evaluation more concretely than most, tells teams to read traces: the real inputs and outputs of the system, as many as they can stand, before building any automated metric. His position is that you cannot stop looking at data, and that the work of looking should be made easy with a purpose-built viewer that puts the input, the output and the surrounding records on one screen.

Reading a hundred traces produces a list of failure modes, and that list is the first draft of the specification. Take a support assistant for an online retailer. After an afternoon of reading, the list might say: cites the wrong policy version, promises a refund the agent has no authority to grant, answers in the wrong language, invents an order number, and gets curt with upset customers. Each item becomes a check with a binary outcome. Husain argues for pass or fail labels rather than 1-to-5 scores, and I agree, for a reason beyond convenience. A five-point scale lets two reviewers disagree about whether a 3 is acceptable and never find out. A binary label forces the decision that the product will force anyway.

An early version of the dataset can live in a spreadsheet. What matters is that every row has an input, a reference to the context the system should have used, the expected behavior in plain words, and the failure mode it exercises. Rows should come from real traffic when possible and from deliberate construction when the real cases are rare: the angry customer, the question the policy does not cover, the order that was never placed.

Check with code first, then with a judge

Checks fall into three tiers of cost. The cheapest are assertions that code can evaluate: the output parses against the schema, the cited document ID appears among the retrieved documents, the SQL the system wrote returns the same rows as the reference query, the reply contains no order number absent from the database. These run in milliseconds, they are deterministic, and they should be the bulk of the suite. Husain calls them unit tests and notes that the pass rate is a product decision, since 100% may not be the right target.

The second tier is a model acting as judge, for criteria that code cannot express, such as whether a reply is polite or whether a summary preserves the meaning of the source. Lianmin Zheng and colleagues studied this in the MT-Bench paper (NeurIPS 2023) and found that a GPT-4 judge agreed with human preferences more than 80% of the time, about as often as humans agree with each other. Their paper also catalogs the weaknesses: judges favor whichever answer comes first, favor longer answers, and favor answers written by themselves, and their reasoning is limited. Practical rules follow directly. Randomize the order in pairwise comparisons. Judge with a model from a different family than the one that generated the answer. Keep the rubric to one criterion per judgment.

Then comes the step teams skip, which is validating the judge against human labels on your own data. A single agreement number is misleading when most outputs are good. Suppose 90% of replies are acceptable. A judge that approves everything agrees with the human labels 90% of the time and catches no failures at all. Report the two rates separately: of the replies humans marked bad, what share did the judge flag, and of the replies humans marked good, what share did it pass. A judge worth using has both numbers high on a few hundred labeled examples, and when it does not, the rubric needs rewriting before the judge is trusted with a release decision.

The third tier is a person, used on a sample, on disagreements between the other tiers, and on every case where the stakes justify it.

Noise is the default

Most eval results in circulation are reported without a sense of their precision, and that is where many bad decisions begin. Take a suite of 200 cases on which a system passes 80%. The standard error is the square root of 0.8 × 0.2 / 200, about 2.8 points, so the 95% interval runs from roughly 74.5% to 85.5%. With 1,000 cases the interval narrows to about plus or minus 2.5 points. A change that moves the headline from 80% to 83% on 200 cases sits inside the noise.

Comparisons between two versions are better handled by pairing: run both on the same cases and look only at the cases where they disagree. Say a new prompt fixes 14 cases that the old one failed and breaks 8 that the old one passed. The net gain is 6 cases out of 200, three points. Under the null hypothesis that the prompts are equally good, the 22 discordant cases would split 14 to 8 or more extreme about 29% of the time (a two-sided exact binomial test). That is a weak basis for shipping, and a team that sees only "80% to 83%" would not know it.

To detect a true difference of three points between 80% and 83% with an unpaired design and 80% power, a back-of-envelope calculation calls for around 2,600 cases per system. Pairing reduces that a lot when the two versions mostly agree, though the saving depends on how often they disagree. Evan Miller's 2024 paper on adding error bars to evals sets out the formulas for standard errors, for comparing two models, and for planning an evaluation, and it is short enough to read in an evening. The takeaway is a habit: every number from an eval should travel with its interval, and every comparison should state how many cases flipped in each direction.

Keep the test set clean

Contamination does not require a leak from the internet. A team can do it to itself. If the same 200 cases guide each prompt revision, the prompt gradually fits those cases, and the score on them rises without any promise about new inputs. This is the oldest problem in machine learning and it applies to prompt engineering unchanged.

The remedy is the one data scientists use for models. Split the cases into a development set that you look at and iterate against, and a held-out set that you run rarely, perhaps before each release, and never read the failures from while tuning. Refresh both from production traffic on a schedule, since the distribution of real inputs moves. Version the dataset, and record the dataset version next to every score, so that a number from March can be compared with a number from September only when they were measured on the same cases.

The GSM1k result adds a caution about vendors' claims. When a model's published numbers come from a benchmark that has been public for years, the question to ask is how much of the benchmark the model has seen. A small private test set in your own domain answers that question better than any amount of reading about the model.

Production closes the loop

An offline suite describes the past. Real users keep inventing inputs that no one on the team imagined, so the suite needs a path for production failures to become test cases. The simplest version is a thumbs-down button, or an escalation to a human agent, that writes the full trace to a review queue. Someone reads the queue weekly, labels the trace, and adds it to the development set if it exposes a failure mode the suite lacks. After a quarter the suite reflects the product as it is used, which is a better description of it than anything written at the start.

Husain's third level, A/B testing, belongs at the end of this chain. Offline evals tell you whether a change is safe to try, and an experiment on live traffic tells you whether the change helps users in ways the cases did not capture: shorter handling time, fewer escalations, more completed purchases. The two answer different questions, and a change should usually pass the first before it earns the second. An offline gain that never shows up in the live metric is information too. It means the suite measures something the customers do not care about, and the cases need to be rewritten.

Synthetic cases can fill gaps, with a caveat. A model can generate hundreds of plausible customer messages in an afternoon, which is useful for coverage of rare categories. But synthetic inputs inherit the generator's habits and tend to be cleaner and more cooperative than real ones, so they should supplement real traces, and the pass rate on them should be reported separately from the pass rate on real cases.

Models retire, and the suite stays

A practical reason to invest comes from the vendors' own calendars. Anthropic's pricing documentation in October 2026 lists several models as retired except on specific cloud platforms: Claude Opus 4 and 4.1, Sonnet 4, Haiku 3.5. Other providers deprecate on similar cycles. A team that built on a retired model has to migrate, and the migration is either a week of running a suite and reading the diff, or a month of anxious manual testing.

The suite is also the means by which a new model is tried. When a provider ships something cheaper, faster or smarter, the question "is it good enough for us?" is answered by running the dev set, reading the discordant cases and checking the interval, which can happen on the day of release. Without the suite, the decision is made on reputation, and reputation was built on someone else's benchmark.

Who owns the suite

An eval set needs an owner, and the owner needs the authority to stop a release. The role resembles that of a test lead more than that of a data scientist. The owner curates cases, retires obsolete ones, adds the failures that production produces, and decides which checks gate a deploy. If failing the suite never blocks anything, the suite will be ignored within a quarter.

A reasonable first month looks like this. In week one, read a hundred real traces and write down the failure modes, with a binary label for each trace. In week two, turn every failure mode that code can detect into an assertion, and run the assertions on every change. In week three, write rubrics for the remaining ones, run a judge on the labeled traces, and compute its two rates against the human labels. In week four, split off a held-out set, put the pass rates with their intervals on a dashboard, and wire the assertions into the deployment pipeline so a regression stops the release. Everything after that is maintenance: new failures become new cases.

Sources

1. Zhang, H., Da, J., Lee, D., et al. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. NeurIPS 2024 Datasets and Benchmarks. arXiv:2405.00332.
2. Singh, S., Nan, Y., Wang, A., et al. (2025). The Leaderboard Illusion. arXiv:2504.20879.
3. Husain, H. (2024). Your AI Product Needs Evals. hamel.dev/blog/posts/evals.
4. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685.
5. Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640.
6. Anthropic. Pricing and model deprecations, Claude developer documentation (retired-model listings as of October 2026). platform.claude.com/docs.