What the Transformer changed was the order of computation

The paper that introduced the Transformer reported two translation results. On the WMT 2014 English-to-German task it scored 28.4 BLEU, more than two points above the best previous results, ensembles included. On English-to-French it reached 41.8 BLEU, a new single-model record. A third sentence in the abstract gets less attention than the scores: the French model trained for three and a half days on eight GPUs, which the authors describe as a small fraction of the training cost of the best models in the literature.

Three and a half days on eight GPUs comes to 672 GPU-hours. Meta's model card for Llama 3.1 405B lists 30.84 million H100 GPU-hours, about 46,000 times as many. The two chips differ by an order of magnitude in speed, so the comparison is crude, but it indicates how far the field travelled along the axis that the 2017 paper opened. That sentence is the part of the paper that aged best. Accuracy gains of a couple of BLEU points are the normal currency of machine translation research, and they have been overtaken many times. A model that trains in days on commodity accelerators, because its structure lets the accelerators stay busy, changed what could be tried. My claim in this essay is that the Transformer's lasting contribution is a rearrangement of the order in which computation happens, and that most of what came afterwards, from pretraining to the scale of current models, was made possible by that rearrangement. It also fixed a cost structure that the field is still working to escape.

Attention was already there

The title of the paper, "Attention Is All You Need," encourages the reading that attention was the discovery. It was not. In 2014 Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio had argued that encoding a whole source sentence into one fixed-length vector was a bottleneck for translation models, and proposed letting the decoder soft-search the source sentence for the parts relevant to each word it produced. Their system matched a state-of-the-art phrase-based translator on English-to-French. Attention then became a standard add-on to recurrent networks, the family of models that reads a sequence one element at a time and carries a hidden state forward.

The Transformer kept attention and discarded the recurrence. Its title was a claim about the second half: the recurrent machinery added nothing that attention did not already supply. That change matters because of what recurrence forces. A recurrent layer computes the state at position t from the state at position t − 1, so a sequence of 2,000 tokens takes 2,000 dependent steps, and no amount of hardware shortens the chain. A GPU or TPU has thousands of arithmetic units that are meant to work at once, and a recurrent network gives them one small matrix operation at a time.

The change was in the dependency graph

Table 1 of the paper compares layer types on three measures. Self-attention has a cost of order n²·d per layer for a sequence of length n and representation width d, against n·d² for a recurrent layer. It needs a constant number of sequential operations, against n for the recurrent layer. And any two positions are connected by a path of constant length, instead of a path whose length grows with their distance. The authors point out that self-attention is the cheaper of the two whenever n is smaller than d, which was true of the sentence-length inputs of 2017 and of models whose d was 512 or 1,024.

The sequential-operations column is the one that mattered for the next decade. With attention, every position in a layer is computed at once, as a handful of large matrix multiplications that map neatly onto hardware built to do exactly that. Training time stops being proportional to sequence length times depth and becomes bounded by throughput. A rough count shows how large the gap is. A recurrent step multiplies a batch of hidden states, say 64 of them, by the layer's weight matrices. A Transformer layer multiplies all 64 sequences of 2,048 tokens in one operation, 131,072 token vectors against the same weights. Each weight fetched from memory does 2,048 times more arithmetic, and that ratio is the difference between an accelerator that waits for memory and one that computes. The same property made it feasible to train on long documents, many documents at a time, and eventually most of the text on the public internet.

The constant path length was a second, quieter benefit. In a recurrent network, information from the start of a paragraph has to survive many state updates to influence the end. In a Transformer, one attention layer lets the last token look directly at the first. That helps with long-range dependencies, and with the optimization problems that had plagued training of deep recurrent networks.

Compute became the variable

Within a year of the paper, two systems showed what the property was good for. GPT (2018) pretrained a Transformer to predict the next token on a large corpus of books and fine-tuned it for individual tasks. BERT (2018) pretrained a bidirectional version by masking words and recovering them. Both used unlabeled text, which exists in nearly unlimited quantity, and both relied on being able to train a large model on a lot of it in reasonable time. The recipe of pretraining at scale followed by adaptation replaced a decade of task-specific architectures.

In 2020 Jared Kaplan and colleagues at OpenAI reported that loss on language modeling falls as a power law in model size, dataset size and training compute, over a range of more than seven orders of magnitude. They also found that architectural details such as width and depth had minimal effects within a wide range. Together those two findings turned a research problem into a budgeting problem. If the shape of the network matters little and the loss follows a smooth curve in compute, then the way to a better model is to spend more, and the architecture that lets you spend most efficiently is the one that wins.

The scale of the change is easy to put numbers on. The base model in the original paper had six layers in each of its encoder and decoder and about 65 million parameters. Meta's Llama 3.1 405B, released seven years later, has 126 layers and 405 billion parameters, about 6,000 times more. The block repeated inside it would be recognizable to the 2017 authors: an attention sublayer and a feed-forward sublayer, each wrapped in a residual connection and a normalization. The refinements are real but incremental. Layer normalization moved to the front of each sublayer. Rotary position embeddings replaced the sinusoidal ones. Grouped-query attention shrank the cache. A gated activation replaced the plain one. A decade of effort went into making the same block bigger and cheaper to run, not into replacing it.

One interface for every task

A second change came with the first and gets less credit. Before 2018, each language task tended to have its own architecture and its own labeled dataset: a tagger here, a question-answering model there. A single Transformer trained to predict text could be pointed at any of them. Colin Raffel and colleagues at Google (T5, 2019) made the idea explicit by recasting every task as text in and text out, so that translation, classification and summarization shared one model and one loss. In 2020 Tom Brown and colleagues at OpenAI showed that a 175-billion-parameter decoder-only model could perform new tasks from a few examples placed in the prompt, with no weight updates at all.

That interface is why language models turned into a platform. A prompt is a program written in the model's own input format, and anyone who can write can try one. The product category of the last three years, from chat assistants to agents, rests on a property of the architecture's training objective and not on any single technical trick. The original paper was a translation paper, with an encoder and a decoder. The decoder-only descendants dropped half of it.

The bill arrives at inference

Parallelism was paid for with quadratic cost. Comparing every position with every other position means an n × n score matrix per head per layer. For a window of 131,072 tokens, that is about 17 billion scores per head per layer, and at each generation step the new token must still be scored against all 131,072 earlier ones in every head and layer. The cache that spares the model from recomputing the past grows linearly with length, for every conversation held in memory. The attention cost that was negligible for 50-word sentences defines the economics of long-context products today. The original authors saw the issue coming. Their conclusion says they planned to investigate local, restricted attention mechanisms to handle large inputs and outputs such as images, audio and video. Nine years on, that sentence reads as a research agenda that is still open.

This is where the Transformer's origin story and its current limits meet. The design traded sequential time in training for quadratic work in sequence length, and in 2017 that was a bargain, since sequences were short and training time was the constraint. Today training runs once and serving runs billions of times, with sequences that are thousands of times longer. The trade that made the architecture win is not the one that now dominates its cost.

What the hybrids concede

The most visible response in 2025 was to keep attention, but use less of it. A family of hybrid models replaces most attention layers with linear-time layers, such as state-space layers in the Mamba family or gated linear-attention variants, which compress the past into a fixed-size state and therefore need no growing cache. IBM's Granite 4.0 models, announced on 2 October 2025, interleave Mamba-2 layers and transformer layers in a 9:1 ratio, and IBM reports a reduction of more than 70% in memory for long inputs and many concurrent sessions. NVIDIA's Nemotron-H replaces the majority of attention layers with Mamba ones. Moonshot's Kimi Linear, a 48-billion-parameter model with 3 billion active, combines a linear-attention layer with latent attention and reports up to 75% less KV cache and up to six times the decoding throughput at a million tokens. Alibaba's Qwen3-Next follows a similar plan. These are vendor-reported figures on vendors' chosen comparisons, and they should be read that way.

The structure of the compromise is more informative than the numbers. Every one of these designs keeps some full attention layers. The usual explanation is that compressing the past into a fixed-size state loses the ability to recall an arbitrary earlier detail exactly, and exact recall is what attention does best, so the designers keep a minority of attention layers to cover it. The result is a model that looks like a Transformer with most of its attention removed. If that is where the architecture settles, the 2017 paper was right about attention and the field is now working out how little of it is enough.

What a buyer should take from this

Most readers will not train an architecture, but its properties arrive in their invoices and their results. Long-context pricing, the memory footprint of concurrent sessions and the quality of recall from very long inputs all trace back to attention's cost structure and to the shortcuts used to reduce it. When a vendor markets a model on its efficiency at long context, the useful questions are the ones about the trade: how much exact recall does the model keep at your input lengths, and does your serving stack have efficient implementations of the new layer types? A savings figure measured on the vendor's stack does not transfer automatically to yours.

The counterargument is that none of this is structural, because scale and data did the work and any sufficiently flexible architecture would have won. The Kaplan finding that depth and width matter little within a wide range supports part of that view. But it does not say that all architectures are interchangeable, only that variations within a family are minor. A family that cannot use the hardware well never reaches the part of the curve where spending more pays off. Saying that scale did the work and saying that the architecture made scale possible are the same observation from two directions.

The next architectural change will probably be judged in the same way, by what it makes cheap. A design that cuts the cost of a million-token window by a factor of four will reshape which products are viable, whatever its scores on benchmarks. The 2017 abstract's third sentence is the model for reading it: look past the headline number to the line about training cost.

Sources

1. Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30. arXiv:1706.03762.
2. Bahdanau, D., Cho, K., Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arXiv:1409.0473.
3. Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
4. IBM (2 October 2025). IBM Granite 4.0: hyper-efficient, high performance hybrid models for enterprise. ibm.com/new/announcements.
5. NVIDIA (2025). Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. arXiv:2504.03624.
6. Kimi Team (2025). Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692.
7. Meta (2024). The Llama 3 Herd of Models. arXiv:2407.21783.
8. Qwen team, Alibaba (September 2025). Qwen3-Next, model announcement (hybrid of gated linear attention and gated attention layers).
9. Raffel, C., Shazeer, N., Roberts, A., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21. arXiv:1910.10683.
10. Brown, T., Mann, B., Ryder, N., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020. arXiv:2005.14165.
11. Meta (2024). Llama 3.1 model card, training time in GPU hours. github.com/meta-llama/llama-models.