Hidden technical debt in machine learning, ten years later
Figure 1 of "Hidden Technical Debt in Machine Learning Systems" is a diagram of boxes. Configuration, data collection, feature extraction, data verification, machine resource management, analysis tools, process management, serving infrastructure and monitoring surround a single small black box labeled ML code. The caption makes the point in a sentence: only a small fraction of real-world ML systems is composed of the ML code, and the surrounding infrastructure is vast and complex. Elsewhere in the paper, D. Sculley and nine Google colleagues put a number on one part of it. A mature system, they wrote, might end up being at most 5% machine learning code and at least 95% glue code.
The paper appeared at NeurIPS in December 2015. Its argument was an uncomfortable one for a field enjoying fast results. Machine learning offers a fantastically powerful toolkit for building useful prediction systems quickly, the abstract begins, and then warns that it is dangerous to think of these quick wins as coming for free. The paper's vocabulary has since become standard for people who run models in production, and ten years on it is worth reading again with a specific question: which of its categories survived the move from classifiers and ranking models to systems built on large language models, which grew, and which have been paid down?
What the paper argued
The central idea is entanglement, and the authors named its consequence the CACE principle: Changing Anything Changes Everything. In a model that combines many signals, no input is truly independent of the others. Adding a feature, removing one, or changing a hyperparameter, a sampling method or a convergence threshold shifts the behavior of the whole, in ways that cannot be isolated. Strict abstraction boundaries, the tool that makes ordinary software maintainable, are eroded because a model's behavior is learned from data and not specified in code.
Several other categories follow from this. Undeclared consumers are systems that quietly start using a model's output, so that changing the model breaks something nobody on the team knew depended on it. Hidden feedback loops arise when a model's predictions influence the data it will later be trained on. Data dependencies cost more than code dependencies because they are harder to analyze: unstable inputs whose meaning shifts under the model's feet, and underutilized ones that add fragility for no gain. Glue code accumulates around general-purpose packages, and pipeline jungles grow where data preparation steps have been added one at a time. Dead experimental codepaths pile up as conditionals in production code that were convenient for a test and never removed. Configuration becomes a form of debt when it is large, fragile and poorly reviewed.
The paper closes with five questions to judge the debt in a system. How easily can an entirely new algorithmic approach be tested at full scale? What is the transitive closure of all data dependencies? How precisely can the impact of a new change be measured? Does improving one model or signal degrade others? How quickly can new team members be brought up to speed? The authors add a caveat worth remembering: moving quickly often introduces debt, and the absence of problems is not evidence of low debt, since the full cost becomes apparent only over time.
Where the interest accrues in LLM systems
Take the five questions to a typical application built on a hosted language model, say a support assistant with retrieval, a few tools and a long system prompt. The categories reappear in new clothes.
Entanglement now has an external trigger. In 2015 the person who changed a feature was on the team. In an LLM application, the base model can change under you. Providers retire models on published schedules, and Anthropic's pricing documentation in October 2026 lists several older models as retired, with availability remaining only on certain cloud platforms. A replacement model with the same name family and nominally the same capabilities will follow instructions slightly differently, format output slightly differently, and refuse different things. Every prompt written against the old model was implicitly tuned to its habits. That is CACE, with the changing agent outside the building.
The prompt is configuration written in natural language, and it carries the worst properties of configuration debt. It is large, since system prompts of several thousand tokens are common. It has no type system. Its diffs are hard to interpret, because a one-word change can alter behavior everywhere and a thirty-line change can alter nothing. And it is rarely reviewed with the rigor given to code, partly because the people editing it include product managers and support leads who are not in the engineering review process. The paper's recommendation for configuration, that a change should be easy to see, easy to assert facts about and reviewed in full, applies unchanged. The tooling to do it is thinner.
Undeclared consumers multiply when the output is text. A pipeline that asks a model to return JSON gets consumers for every field. A dashboard counts how often replies contain a certain phrase. A script downstream splits a reply on a newline and reads the third line. None of them were declared as dependents of the model's formatting, and all of them break when it changes. The remedy is old: put a contract on the output, with a schema, enforced by structured-output features where the provider offers them and validated in code where it does not.
Feedback loops run through generated text. If an assistant's answers are saved as articles in the knowledge base that the same assistant retrieves from, errors reinforce themselves: a wrong answer, once indexed, is retrieved as evidence for the next. The model-collapse literature, such as the 2023 study by Ilia Shumailov and colleagues showing that training on generated data causes the tails of a distribution to disappear, is about training. The same dynamic operates at retrieval time on a smaller scale, and nothing in a typical architecture marks which documents in the index were written by the system.
The glue-code ratio has probably gone the other way. When the model is an API call, the "ML code" in the black box is a few lines and almost everything else is glue: prompt assembly, retrieval, output parsing, retries, caching, tool wrappers, evaluation scripts. Agent frameworks add a layer that wraps those, which the paper would have identified as a general-purpose package whose assumptions get locked in. Its advice, to wrap black-box packages in common APIs so that they can be replaced, is still the best available.
Dead experimental codepaths are the easiest to underestimate. A prompt variant behind a flag, a second retrieval strategy under test, a model fallback that was added for an incident: each is cheap to add and expensive to retire, because nobody remembers what the flag does. The most vivid cautionary tale comes from outside machine learning. On 1 August 2012 Knight Capital's trading systems sent millions of unintended orders in about 45 minutes, which per its own SEC filing cost approximately $461 million. According to the SEC's later order, as summarized in secondary accounts, an old order-routing function unused since 2003 had never been removed, a flag was repurposed to trigger new code, and one of eight servers had not received the update, so the flag woke the old function there. A branch nobody had cleaned up, plus a configuration switch whose meaning had changed, ended Knight's run as an independent firm.
The paper's monitoring advice still works
The section on changes in the external world offers monitoring advice that carries over almost unchanged. The authors suggested comparing the distribution of predicted labels with the distribution of observed ones, which they called a prediction bias check. It is far from a complete test, since a model that always predicts the average passes it, but it is cheap and it catches drift quickly. The LLM version is a set of distribution checks on the outputs: the share of replies that refuse, the median and 95th-percentile length, the share that fail schema validation, the number of tool calls per conversation, the share that cite a source. None of these says whether an answer is correct. Each moves when something upstream has changed, whether a provider updated a model, a document in the index was replaced or a prompt edit went out, and a plot of them over weeks gives an early warning that a quality evaluation, which runs less often, will confirm later.
The second piece of advice is the action limit. Where a system takes actions in the world, such as sending messages or issuing credits, the authors recommended setting broad limits that would not trigger spuriously, with automatic alerts and manual intervention when one is reached. An assistant that can issue refunds should have a ceiling on the amount per refund and on the number per hour, enforced outside the model, and a breach should page a person. The limit does not make the system correct. It makes the worst case bounded, which is what lets a team sleep while the model changes behavior underneath it.
A last item is cultural, and the paper names it. It observes that the split between researchers who build models and engineers who run them creates friction, and proposes embedding the two on the same teams. The LLM version is the gap between the people who write prompts, who are often product or domain specialists, and the people who maintain the systems around them. A prompt that a specialist edits in a web console, without review or tests, is a production change by another name. Whatever process governs code should govern the prompt.
What has been paid down
Not everything got worse. The tooling that the 2015 paper wished for now exists in some form: experiment trackers that record which data, code and settings produced a model, feature stores that give training and serving the same definition of a feature, data validation libraries that check inputs against expectations, model registries that record what is deployed. In 2017 Eric Breck, Sculley and colleagues published the ML Test Score, a rubric of 28 specific tests and monitoring needs for judging production readiness. A team that adopts any serious checklist of this kind has made its debt visible, which is the first step toward managing it.
For LLM systems the equivalent is the evaluation suite: a fixed set of cases that run on every change to the prompt, the retrieval index or the model. It plays the role of unit tests for a system whose behavior cannot be specified in code, and it answers the third and fourth of the paper's questions directly, which are about measuring the impact of a change and noticing when improving one thing degrades another.
The five questions, rewritten
The paper's closing questions translate well. The right column is what a team should be able to answer in a review, with evidence.
| Original question | In an LLM system | A good answer looks like |
|---|---|---|
| How easily can a new algorithmic approach be tested at full scale? | Can we swap the model, or the retrieval method, and run the full evaluation in a day? | A script and a dashboard, not a project |
| What is the transitive closure of data dependencies? | Which documents, tools and APIs can influence an answer, and who owns each? | A list that someone keeps current |
| How precisely can the impact of a change be measured? | Does a prompt edit come with before-and-after scores and an interval? | Yes, on every pull request |
| Does improving one model or signal degrade others? | Does a fix for one failure mode regress another? | The suite is sliced by failure mode |
| How quickly can new members get up to speed? | Can a new engineer explain why the system prompt says what it says? | Each instruction has a recorded reason |
The last row is the least technical and the most revealing. A system prompt in which every line exists because of an incident and nobody remembers which one is a body of rules without provenance, and rules without provenance cannot be safely deleted. Attaching a short reason and a test case to each instruction is cheap at the time of writing and very expensive to reconstruct later.
Debt is a loan, not a sin
The framing of debt has a flattering reading and an unflattering one. The flattering one is that taking on debt to ship sooner is sometimes the right call. A prototype that proves demand before a quarter ends can justify a messy prompt and a pile of glue. The unflattering one is that most debt is not chosen. It accumulates through decisions that each looked reasonable, and its cost is invisible until a model retirement, an audit or an incident makes it concrete.
What separates a loan from a trap is whether the debt is visible and owned. A team can answer the five questions above for its largest LLM system in an hour. The questions it cannot answer are the debt, and writing them down with a name next to each is the cheapest piece of risk management available. The hour is worth scheduling for the next quiet afternoon.
Sources
- Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28.
- Breck, E., Cai, S., Nielsen, E., Salib, M., Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE Big Data 2017.
- Anthropic. Pricing and model deprecations, Claude developer documentation (retired-model listings as of October 2026). platform.claude.com/docs/en/about-claude/pricing.
- Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., Anderson, R. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493.
- Knight Capital Group (2012). Form 8-K and related filings with the U.S. Securities and Exchange Commission, August 2012; U.S. Securities and Exchange Commission (2013), Order in the matter of Knight Capital Americas LLC.