Scaling laws forecast loss, and the forecast changed twice

In March 2022 a group at DeepMind led by Jordan Hoffmann released a model called Chinchilla. It had 70 billion parameters, and it was trained with the same compute budget as Gopher, DeepMind's own 280-billion-parameter model from three months earlier. Chinchilla beat Gopher on a wide range of tasks, and also beat GPT-3 (175 billion), Jurassic-1 (178 billion) and Megatron-Turing NLG (530 billion). It reached 67.5% on the MMLU benchmark, more than seven points above Gopher. The way it did so was unglamorous. It was a smaller model trained on about four times as much text.

The paper's conclusion was blunt. Large language models of the time were significantly undertrained, because the field had been scaling up parameters while holding the amount of training data roughly constant. That was not a quirk of one lab. It followed the best guidance available, the scaling-law study that Jared Kaplan and colleagues at OpenAI had published two years earlier, which had suggested that with a growing budget most of the extra compute should go into bigger models. A recipe that had guided the most expensive experiments in the field turned out to be misallocating them by a large factor, and it took two years and one expensive counterexample to find out.

I start there because the phrase "scaling laws" carries more authority than the evidence warrants. They are empirical curve fits. They are useful, and they have been revised at least twice, once about how to split a budget and once, now in progress, about whether the data to follow the curve exists. Anyone making plans that depend on what models will be able to do in two years should know what the curves actually forecast and where the revisions are coming from.

What a scaling law forecasts

The Kaplan study found that the loss of a language model, which is its average error in predicting the next token on held-out text, falls as a power law in the number of parameters, the size of the dataset and the amount of training compute, over a range spanning more than seven orders of magnitude. It also found that width and depth had minimal effects within a wide range. A power law is a straight line on a log-log plot, which means the relationship can be fitted from many small and cheap runs and extended to a large and expensive one.

That is the practical use. A lab runs dozens of small models, fits the curve, and predicts the loss of a run that will cost tens of millions of dollars before spending the money. The forecast is quantitative and has a good track record for the thing it measures.

The thing it measures is loss, and loss is not usefulness. A model's score on a downstream task depends on loss in a way that can look abrupt: a capability seems to appear suddenly at a certain scale. Rylan Schaeffer, Brando Miranda and Sanmi Koyejo argued at NeurIPS 2023 that many of these apparent jumps come from the choice of metric. An all-or-nothing score such as exact-match accuracy on a multi-step problem stays near zero while the underlying per-token ability improves smoothly, then rises sharply once every step is likely to be right. Switch to a metric that gives partial credit and the jump becomes a slope. The implication for planning is that scaling laws forecast the smooth quantity well and the thing a business cares about, whether the model can do task X reliably, only indirectly.

The exponents show how demanding the curve is. In Kaplan's fit, loss falls with the number of non-embedding parameters N as N to the power of minus 0.076. Doubling the parameters therefore multiplies the loss by 2^-0.076, about 0.949, a 5% improvement. A tenfold increase gives 10^-0.076, about 0.839, a 16% improvement. Since a tenfold increase in parameters at fixed data costs roughly ten times the compute, each further 16% asks for another factor of ten in spending. The curve has no cliff and no finish line, only a steadily rising price, and underneath it sits a floor set by the unpredictability of language itself. That is a very different picture from the one in which "scaling" means that more of everything produces proportionally more intelligence. It also explains why the capital intensity of frontier training rose so sharply: a smooth, favorable curve whose cost grows geometrically rewards whoever can finance the next step.

The arithmetic of compute-optimal training

The training compute of a dense transformer is roughly six floating-point operations per parameter per training token, so C ≈ 6·N·D, where N is the parameter count and D the number of tokens. For a fixed C, a larger N means a smaller D. Hoffmann's team asked what split minimizes loss, and their answer was equal proportions: for every doubling of model size, double the number of training tokens.

Chinchilla itself shows the effect. It used 70 billion parameters and about 1.4 trillion tokens, so 6 × 70 × 10⁹ × 1.4 × 10¹² is about 5.9 × 10²³ operations. Gopher used 280 billion parameters and 300 billion tokens, 6 × 280 × 10⁹ × 300 × 10⁹, about 5.0 × 10²³. The budgets were of the same order, and the two models differed by a factor of four in size and by almost five in data. The ratio Chinchilla landed on, about 20 tokens per parameter, became the field's rule of thumb for what "compute-optimal" means.

Optimal for training is not optimal for serving

The rule has a catch that matters to anyone paying for inference. Compute-optimal minimizes the cost of the training run for a given loss. A model is trained once and served for years, and serving cost grows with parameter count. A lab that expects heavy use should therefore train a smaller model on far more tokens than Chinchilla's ratio, accepting a less efficient training run in exchange for a cheaper product. Nikhil Sardana and colleagues at MosaicML and Databricks put numbers on this in a 2024 ICML paper, by adding an inference term to the Chinchilla framework. For a lab that expects demand on the order of a billion requests, they concluded the model should be trained smaller and longer than Chinchilla-optimal, and the larger the expected demand, the further the optimum moves from the original rule.

Meta's Llama 3 models are the clearest example. The 8-billion-parameter model was trained on over 15 trillion tokens, about 1,900 tokens per parameter, close to a hundred times the Chinchilla ratio. Its training compute, 6 × 8 × 10⁹ × 15 × 10¹², comes to about 7.2 × 10²³ operations, similar to Chinchilla's. The same budget bought a model roughly a ninth the size to serve. Whether it reaches the quality of a 70-billion-parameter model depends on the task and the training recipe, and the comparison is not between like systems, but the direction of the trade is clear. Money spent on training tokens is paid once. Money saved on parameters is saved on every request.

This explains something about the shape of the product catalogs that vendors offer. Small and mid-sized models in current families are often much better than their parameter counts would have suggested in 2022, because they were trained far past the point that the older rule recommended. For a buyer, it is one reason to test the cheaper tier before assuming the largest model is necessary. It is also a reason to be skeptical of any comparison that uses parameter count as a proxy for quality: two models of the same size can differ by a factor of ten or more in the tokens they saw, and the number of tokens is rarely on the price list.

The data wall

Training long on a small model requires large amounts of text, and the stock of human-written text is finite. Pablo Villalobos and colleagues at Epoch AI estimated in 2022, with an update in 2024, how much usable public text exists. Their 2024 figure for the effective stock, counting only text good enough for training and allowing for repeated passes, is roughly 300 trillion tokens, with a 90% interval from 100 trillion to 1,000 trillion. Under then-current trends, they projected that models would be trained on datasets about equal to that stock between 2026 and 2032, and earlier if models are heavily overtrained. It is now October 2026, so the early part of that window has arrived.

Context helps. Llama 3's 15 trillion tokens is about a twentieth of the 300 trillion estimate. Compute for frontier training, by Epoch's count, has grown about four to five times per year, and if data requirements track compute at anything like that pace, a twentieth does not last long: at 4.5 times a year, a twentyfold increase takes two years. Compute can keep growing as long as money and power hold out. Public human text cannot.

The ways around the wall each come with a catch.

Repetition is the cheapest. Niklas Muennighoff and colleagues found that training for up to four passes over the same data produced negligible changes in loss compared with fresh data, and that beyond that the value of extra compute decays toward zero. This stretches the stock by a small multiple, and Epoch's effective figure already allows for it.

Synthetic data, text generated by models, can in principle supply any amount. But Ilia Shumailov and colleagues showed, in a study later published in Nature, that training repeatedly on model-generated content causes the tails of the original distribution to disappear, which they called model collapse. The result is for recursive training on a model's own output without fresh human data, and the practical situation differs, since labs filter and mix. Still, it marks a limit: generated text can only repeat and recombine what the generator already contains. It works best where correctness can be checked automatically, as in mathematics and code, where a solver or a test suite can discard the wrong samples.

Private and licensed data is a third source, and it changes who holds bargaining power. Text behind paywalls, inside companies, or in formats that need heavy processing sits outside the public stock Epoch counted, which is one reason access agreements and data partnerships matter to a lab's plans.

The fourth is to stop putting most of the extra compute into pretraining. Reinforcement learning on verifiable tasks and extra computation at inference both turn compute into capability without requiring more human text. The reasoning models of 2024 and 2025 are the clearest large instance of this move, and it is part of why the frontier changed shape.

Reading a scaling claim

A few questions make vendor claims and analyst forecasts easier to evaluate.

1. What is on the vertical axis? Loss on held-out text, a benchmark score, or something that resembles your task? A smooth curve on loss says little about when a specific capability becomes reliable.
2. How far beyond the measured range does the line extend? Fitting the curve across seven orders of magnitude is a different claim from drawing it one more order beyond anything observed.
3. What was held fixed? Curves that scale parameters with fixed data, or compute with fixed data, miss the Chinchilla effect.
4. Which cost is being minimized, training or serving? A model optimal by one criterion can be poor by the other.
5. Is the extrapolation consistent with the data supply? A forecast that assumes tokens keep growing at today's pace should say where they come from.

None of this says progress will stop. It says the sources of progress change, and the labs' mix of pretraining, post-training and inference compute appears to have shifted to reflect it.

Build your own curve

The most reliable forecast a company can have is one it makes itself. Every release of a model from every vendor you use can be run through the same evaluation suite, with the score recorded next to the release date and the price. After a year, those points form a curve for your task, with the performance you got per dollar and per second, and the trend is a basis for planning that does not depend on any lab's chart.

It also changes the questions you ask vendors. Instead of asking whether scaling will continue, ask what lever is expected to deliver the next gain, and whether that lever raises your costs (more inference compute per answer) or lowers them (a cheaper model of the same quality). The answer determines whether next year's capability arrives with a smaller bill or a larger one.

Sources

1. Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.
2. Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
3. Schaeffer, R., Miranda, B., Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? NeurIPS 2023. arXiv:2304.15004.
4. Meta (April 2024). Introducing Meta Llama 3: the most capable openly available LLM to date. ai.meta.com/blog/meta-llama-3.
5. Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., Hobbhahn, M. (2022, revised 2024). Will we run out of data? Limits of LLM scaling based on human-generated data. arXiv:2211.04325.
6. Epoch AI (2024). Training compute of frontier AI models grows by 4-5x per year. epoch.ai/blog.
7. Muennighoff, N., Rush, A. M., Barak, B., et al. (2023). Scaling Data-Constrained Language Models. arXiv:2305.16264.
8. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., Anderson, R. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493; and AI models collapse when trained on recursively generated data, Nature 631 (2024).
9. Sardana, N., Portes, J., Doubov, S., Frankle, J. (2024). Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. ICML 2024. arXiv:2401.00448.