Scaffolding around a model is a bet against the next model

On 13 March 2019 Rich Sutton published a short essay, "The Bitter Lesson," that opens with a sentence few researchers have been able to ignore: the biggest lesson from seventy years of AI research is that general methods which use ever more computation win in the end, and by a wide margin. He supports it with four histories. Chess programs that beat Kasparov in 1997 did it with massive deep search, to the dismay of researchers who had built in human understanding of the game. Computer Go followed the same arc about twenty years later, when search combined with learning by self-play made the hand-crafted knowledge irrelevant or worse. In speech recognition, statistical methods that used more computation beat systems built on knowledge of phonemes and the vocal tract, and deep learning later reduced the human input further. In vision, edge detectors and hand-designed features gave way to networks that use only convolution and a few invariances.

Sutton's account of why the mistake keeps recurring is the part most often skipped. Building knowledge into a system, he says, always helps in the short term and is personally satisfying to the builder. In the long run it plateaus, and it complicates the method in ways that make it harder to benefit when more computation arrives. The temptation is real, and it is not limited to research labs. It is the standing temptation of anyone who builds on top of a model: write the clever prompt, add the validation step, split the task into five calls, encode the rules a human expert would use. Each of those is a small piece of built-in knowledge. Each is a bet that the next model will not make it unnecessary.

Sutton is more careful than the slogan version of his essay. He says that leaning on human knowledge and leaning on computation need not conflict, though in practice they do: time spent on one is time not spent on the other, and the knowledge-heavy approach tends to complicate a method in ways that stop it from absorbing the next increase in computation. That is a claim about opportunity cost and about coupling. It leaves room for knowledge that the system cannot learn on its own, and the rest of this essay depends on that room.

I want to argue that the bet is worth pricing explicitly. Some scaffolding around models pays for years. Some expires within a model generation, and the cost of keeping it is paid in complexity, latency and tokens long after it stopped helping. The practical skill is telling the two kinds apart in advance.

A technique that expired in three years

The cleanest case is chain-of-thought prompting. In January 2022 Jason Wei and colleagues at Google showed that prompting a 540-billion-parameter model with eight worked examples, each including its reasoning steps, reached state-of-the-art accuracy on GSM8K, the grade-school math benchmark, ahead of a fine-tuned GPT-3 with a verifier. Months later Takeshi Kojima and colleagues found that even the examples were unnecessary. Appending "Let's think step by step" to a question took InstructGPT's accuracy on MultiArith from 17.7% to 78.7% and on GSM8K from 10.4% to 40.7%. A sixty-one-point gain from one sentence was one of the stranger results of that period, and the phrase became a staple of prompting guides.

Then labs started training models to do it themselves. OpenAI's o1, released in September 2024, was trained with reinforcement learning to produce a long chain of reasoning before answering. DeepSeek's R1, published in January 2025 and later in Nature, showed that reasoning behavior could be incentivized through reinforcement learning on tasks with checkable answers, without human-labeled reasoning traces, and that the resulting model outperformed models trained on human demonstrations on verifiable tasks in math, coding and science. The technique that users had to request was now something the model did unprompted.

Asking for it anyway turned out to have a price. In June 2025 Lennart Meincke, Ethan Mollick, Lilach Mollick and Dan Shapiro at Wharton published a report on chain-of-thought prompting across several current models. For reasoning models the gains in accuracy were marginal, often nothing, while the added time ran from 20% to 80%. For some non-reasoning models there was a modest improvement, with more variance in the answers, including occasional errors on questions that the model would otherwise have gotten right. A technique that was worth sixty points in 2022 was, three years later, a tax on latency for the models that mattered most.

The prompting guides written in 2022 did not become wrong because their authors erred. They became wrong because the capability they were compensating for moved into the weights. That is Sutton's argument in miniature, played out over three years on a technique anyone could apply in one line.

Test-time compute is the same lesson aimed at inference

The reasoning models are an instance of the lesson as well as a casualty of it. Sutton named two classes of method that scale with computation: search and learning. For most of the history of language models, learning happened at training time and a call to the model was a single forward pass. A line of work in 2024 asked what would happen if computation were also spent at the moment of answering. Charlie Snell and colleagues reported in August 2024 that, for problems on which a smaller model has a non-trivial success rate, spending extra computation at test time could outperform a model fourteen times larger in a FLOPs-matched comparison, and that allocating that computation adaptively improved efficiency more than fourfold over a best-of-N baseline.

For builders the implication is awkward. A great deal of application engineering from 2023 and 2024 consisted of manual versions of test-time search: sample five answers and take the majority, ask the model to critique and revise its own draft, break a problem into sub-questions and aggregate. Some of this is now built in, trained into models that decide for themselves how long to think. The elaborate multi-call pipeline that outperformed a single call two years ago may be outperformed by a single call to a newer model, at lower cost and with fewer places to fail.

The pull of the clever fix

Sutton's account of the bitter lesson includes a claim about psychology, and it fits product teams as well as it fits researchers. The fix that works around a model's failure is concrete. It shows up in a demo the same afternoon. The person who wrote it understands exactly why it works, which is a satisfaction that "wait for the next model" cannot offer. Organizations add a second force: a scaffold, once built, acquires an owner, a repository and a line on a roadmap, and deleting it feels like criticizing someone's work. The cost of keeping it is spread thin across latency, tokens and fragility, so nobody receives a bill with its name on it.

The remedy is to make deletion routine instead of personal. A scaffold that enters the codebase with a stated purpose, a measured benefit and a review date can leave when the review says so, and nobody has been judged. That sounds like bureaucracy, but it is the same discipline as a feature flag with an expiry.

Three kinds of scaffolding

Not all structure around a model is the same, and sorting it is the useful exercise. One kind compensates for a weakness of the model: it can't count reliably, it forgets instructions in a long prompt, it loses the thread of a multi-step problem, it formats output inconsistently. The other kind encodes knowledge about your world that the model could not have: your product catalog, your refund policy, your database schema, what your regulators require, which customers are protected by contract terms. The first kind is perishable by construction, since model builders are working on precisely those weaknesses. The second kind does not depend on the model's progress, and it becomes more valuable when the model gets better at using it. A third kind exists for assurance and has nothing to do with capability: the approval step before a refund above a set amount, the permission check on a database write, the audit log of every tool call. A smarter model does not retire those, because they exist so that a person or a regulator can answer for what the system did.

A ledger makes the distinction concrete. Consider a document-classification pipeline built in 2024 and examine its parts.

Piece	What it compensates for	Evidence it helps	Retest when
"Think step by step" in the prompt	Weak multi-step reasoning	Accuracy with and without, on your cases	Every model change
Five-call chain (extract, normalize, classify, verify, format)	Errors when one prompt does everything	Per-step accuracy against the single-call baseline	Every model change
Retry loop with a regex check on output	Unreliable formatting	Share of outputs that fail the check	Every model change
Twelve-rule table mapping vendors to ledger categories	Nothing: this is company policy	It is the specification	When policy changes
Retrieval of the customer's contract terms	Model lacks private data	Answers cite the right clause	When the contract store changes
Human approval for credits above a set amount	Nothing: this is accountability	Required by policy and audit	When the policy changes

The first three rows are bets against the next model. The last three are not, and a stronger model makes the middle one better. Effort tends to flow toward the rows that will expire and away from the ones that last, because the perishable pieces are the ones that produce visible improvement quickly.

The cost of carrying a perishable piece can be worked out. Suppose the five-call chain in the table takes 1.2 seconds per call, so a document takes 6 seconds, and each call resends the 3,000-token document. That is 15,000 input tokens per document against 3,000 for a single call. At an illustrative $2 per million input tokens, the chain costs $0.030 per document and the single call $0.006. Across 200,000 documents a month the difference is $4,800, plus a wait that is four times as long. If a newer model does the whole job in one call with the same accuracy, that difference is a pure tax for keeping the old design.

Where the lesson may not apply

There are fair objections. Sutton wrote about research programs measured over decades, and a product team ships on a quarter. A workaround that works today and costs a day to remove later is a sound engineering decision. The lesson argues for pricing the removal and not for refusing to build.

A second objection is that the premise of ever-cheaper computation faces limits. Test-time compute is not free: reasoning models can produce thousands of hidden tokens per answer, billed as output, which can make a call several times more expensive than a plain one. Where a task is high-volume and simple, a small model with a precise prompt and a validator may remain the right engineering answer for years, because the stronger approach costs more than the task is worth.

A third is that Sutton himself does not believe the current paradigm completes the story. In a September 2025 interview with Dwarkesh Patel he argued, as Patel summarized it, that language models learn to predict what people would say instead of what happens in the world, and that the field will need systems that learn continually from experience. With David Silver he had made a related argument in an April 2025 paper on an "era of experience." If he is right, the next large change to application design could come from a direction that today's scaffolding does not anticipate. The lesson supports humility here as well: bets on a particular architecture can age as fast as bets on a particular prompt.

Run the upgrade-day test

The practical habit is cheap. Every time a new model arrives, run your evaluation suite twice, once with the scaffolding in place and once with the perishable pieces switched off: the step-by-step instruction, the extra verification call, the retry loop, the multi-call chain collapsed into a single call. Compare accuracy, latency and cost per task, with intervals. Where the scaffolded version no longer wins, delete it. Where it still wins, record by how much and set the next review for the next model.

Over a few release cycles the ledger shows which of your decisions were durable. The pieces that encode your company's rules, data and obligations keep their value through every upgrade, and those are the ones that deserve the investment, the documentation and the owners. The pieces that compensate for the model's weaknesses will mostly disappear, and the savings from deleting them arrive without any new engineering.

Sources

1. Sutton, R. (13 March 2019). The Bitter Lesson. incompleteideas.net/IncIdeas/BitterLesson.html.
2. Wei, J., Wang, X., Schuurmans, D., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arXiv:2201.11903.
3. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., Iwasawa, Y. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS 2022. arXiv:2205.11916.
4. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature 645, 633-638. arXiv:2501.12948.
5. Meincke, L., Mollick, E., Mollick, L., Shapiro, D. (2025). Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting. Wharton Generative AI Labs. arXiv:2506.07142.
6. Snell, C., Lee, J., Xu, K., Kumar, A. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv:2408.03314.
7. Patel, D. (September 2025). Richard Sutton, Father of RL, Thinks LLMs Are a Dead End. Dwarkesh Podcast. dwarkesh.com/p/richard-sutton.
8. Silver, D., Sutton, R. S. (2025). Welcome to the Era of Experience. Preprint, April 2025.