Inference economics shapes the product you can afford to build

Take Llama 3.1 70B, a model whose configuration is public: 80 layers, 8 key-value heads, 128 dimensions per head. For every token a conversation holds, the server keeps a key vector and a value vector in every layer. At 16-bit precision that comes to 2 × 80 × 8 × 128 × 2 bytes, or 327,680 bytes per token. Fill the model's 128,000-token window and one conversation occupies about 43 gigabytes, roughly 40 GiB, of GPU memory. The weights of the model, at 16 bits per parameter, take 140 GB. A single user with a long document open is holding about as much memory as a third of the model itself.

I start with that figure because it turns the price sheet of an AI provider into something legible. The prices look arbitrary until you see the hardware constraint underneath: output tokens cost several times what input tokens cost, cached tokens cost a fraction, batch jobs cost half, and some providers charge more per token once a prompt passes a length threshold. All of those follow from two facts about how transformers are served. A team that understands them can design a product that is cheap to run. A team that treats the model as a black box with a per-token fee learns what its design costs from the first invoice.

Decoding waits on memory, not arithmetic

Serving a request has two phases. In prefill, the model reads the whole prompt at once and builds the key-value cache. The tokens are processed in parallel, so the work is dense matrix multiplication and the chip is limited by its arithmetic speed. In decoding, the model emits one token at a time, each depending on the one before it, and every step has to stream the model's weights from memory to the compute units.

The consequence shows up in a calculation anyone can redo. An H100 (SXM variant) has about 3.35 terabytes per second of memory bandwidth and about 989 teraflops of dense BF16 compute. A 70B-parameter model needs roughly 140 gigaflops per generated token (two operations per parameter) and, at 16 bits, 140 GB of weights to read per step. Pretend for a moment that a single GPU held the whole model. Compute alone would allow about 7,000 tokens per second. Memory bandwidth alone would allow about 24 steps per second, because each step must read 140 GB.

If a step serves one conversation, the chip produces 24 tokens per second and uses well under one percent of its arithmetic capacity. If a step serves 300 conversations at once, the same weight read produces 300 tokens, and compute and bandwidth are about equally loaded. Sharding across several GPUs divides both numbers but leaves the ratio alone. The ridge sits near 300 concurrent sequences. Below that, the hardware is waiting on memory, and the cost of each token is mostly the cost of sharing a weight read among too few people.

This is the reason serving is a batching problem, and the reason providers can sell tokens at all. It also explains a latency floor that no software trick removes: an eight-GPU node holding 17.5 GB of weights per card must still read that much per step, which takes about 5 milliseconds at 3.35 TB/s. Generating a 500-token answer takes at least two and a half seconds of decoding even when the system is otherwise idle. Prefill is different. A long prompt costs compute but arrives in one parallel pass.

The cache is memory rented per conversation

Batching has a limit, and the limit is the KV cache. Every active conversation needs its own cache, sized at 327,680 bytes per token for the model above. At 8,000 tokens, that is 2.7 GB. At 128,000 tokens it is 43 GB.

Take an illustrative node of eight 80-GB GPUs. The weights use 140 GB, leaving roughly 500 GB for caches before overheads. That is room for about 185 conversations of 8,000 tokens, or about 11 conversations of 128,000 tokens. One long-context user takes the place of sixteen short ones. A flat per-token price hides this, which is why some providers have started to charge for it directly. Anthropic's list price for Claude Haiku 5.5, as of October 2026, is $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, and five times that for prompts above the threshold. The surcharge is the provider's way of passing on the memory a long prompt holds.

The way caches are managed matters as much as their size. Woosuk Kwon and colleagues, mostly at UC Berkeley, argued in the vLLM paper (SOSP 2023) that serving systems of the time wasted cache memory through fragmentation and duplication, which limited batch sizes. Their PagedAttention method, which stores the cache in fixed-size blocks the way an operating system pages memory, improved throughput 2 to 4 times at the same latency on the systems they compared against. A gain of that size from memory management alone says how binding the constraint is.

Model designers attack the same constraint from the other side, and their choices become your cost structure. Grouped-query attention, introduced by Joshua Ainslie and colleagues at Google in 2023, lets several query heads share one key-value head. Llama 3.1 70B has 64 query heads but only 8 key-value heads. Had it kept a separate pair for every head, the cache would cost about 2.6 MB per token instead of 0.33 MB, and a full 128,000-token conversation would hold roughly 340 GB. Speculative decoding, from Yaniv Leviathan, Matan Kalman and Yossi Matias at Google, goes after the sequential bottleneck instead: a small draft model proposes several tokens, the large model checks them in a single parallel pass and keeps the ones it agrees with, and the output is identical to ordinary decoding. They reported speedups of two to three times on T5-XXL. Quantization stores weights and caches in 8 or 4 bits, which shrinks both the memory read per step and the memory held per conversation, at a quality cost that has to be measured on your own task. A buyer meets these techniques as lower prices or faster responses. A team that hosts its own models gets to choose among them.

A price list is physics in dollars

Anthropic's current price list makes a consistent pattern visible. Across its models in October 2026, output tokens cost five times what input tokens cost: Haiku 5.5 at $0.10 and $0.50, Sonnet 5.5 at $2 and $10, Opus 5.5 at $4 and $20. Reading from a cache costs a tenth of the input price on most models and 5% or less on several of the newest. Batch requests carry a 50% discount on both. At launch in August 2025, OpenAI listed GPT-5 at $1.25 input and $10 output per million tokens, an eightfold gap, so the exact ratio varies by vendor while the direction does not.

Providers do not publish their cost bases, so what follows is my reading rather than a statement of theirs. The output premium matches the decoding bottleneck: each output token is a sequential step that occupies a slot in the batch and rereads the weights. The cache discount matches the fact that a cached prefix has already paid for its prefill, and what remains is memory rental. The batch discount matches the opportunity to fill idle capacity at off-peak hours when nobody is waiting for the answer. Seen this way, the list tells a product team where the provider's costs are low and where they are high, which is what a good designer wants to know.

Reasoning models sharpen the output premium. The thinking tokens a model produces before answering are billed as output. A request that returns a 100-token answer after 5,000 tokens of reasoning is billed for 5,100 output tokens, and the bill is dominated by text the user never sees.

What a 6,000-token system prompt costs

Here is a constructed support assistant, priced at Sonnet 5.5's list: $2 per million input tokens, $10 per million output tokens, $2.50 per million to write a prefix into the 5-minute cache and $0.10 per million to read from it. The system prompt and policy text come to 6,000 tokens. A conversation has 8 turns. Each user message is 300 tokens and each reply 250.

Input grows with every turn, because the history is resent. Turn 1 sends 6,300 tokens, turn 2 sends 6,850, and so on up to 10,150 on turn 8. Summed over the eight turns, the input is 65,800 tokens. The output is 2,000 tokens. Without caching, the input costs $0.1316 and the output $0.0200, so a conversation costs about $0.152. At 10,000 conversations a day that is roughly $1,500 a day.

Now cache the 6,000-token prefix only. The first turn writes it ($0.0150), seven later turns read it (7 × 6,000 tokens × $0.10 per million, about $0.0042), and the remaining 17,800 tokens of input are billed at the normal rate ($0.0356). With the unchanged output cost, the conversation costs about $0.075, roughly half. Nothing about the model or the quality changed. The saving came from putting the static material first and keeping it byte-identical from one request to the next.

The same arithmetic shows something about output. The 2,000 output tokens are 3% of all tokens in the conversation and 27% of the cached bill. Shortening replies by a third, with a tighter format or a stricter instruction, cuts the conversation cost by about 9%. Cutting a third from the cached system prompt, 2,000 tokens of carefully written policy, saves about the same and puts at risk the behavior that text exists to produce. Trimming the prompt is the first thing most people reach for, and the numbers say the reply is the safer target.

These calculations suggest a handful of practices. Keep the stable parts of the prompt at the front and the volatile parts at the end, since changing a single early token invalidates everything after it. Version the system prompt on purpose, and treat each edit as having a cache cost. Summarizing old turns saves tokens but rewrites the prefix, so measure the trade rather than assuming. Route non-interactive work, such as nightly classification or document enrichment, to batch endpoints at half price. Limit context length by product tier instead of leaving it open, because the memory a conversation holds is the least visible way it costs money.

Falling prices do not retire the design problem

The obvious objection is that none of this will last, because token prices fall quickly. They do. Epoch AI estimates that the price of reaching a fixed level of performance falls by roughly 40 times a year, with rates from 9 times to 900 times depending on the benchmark and the performance threshold. Stanford's 2025 AI Index, working from Epoch's data, reports the same range. An architecture tuned to today's list may look cramped next year.

Three things limit the comfort. First, the decline is measured at a fixed capability. Teams usually want the best model available, and the flagship tier's price is not falling at 40 times a year: Anthropic's top tier lists $10 and $50 per million tokens. Second, designs consume what they are given. Anthropic reports that agents use about four times the tokens of a chat and multi-agent systems about fifteen times, which means cheaper tokens get spent on more steps. Third, Epoch itself cautions that the fastest declines occurred in the most recent year and may not persist.

More useful than any price level is the structure. Outputs cost a multiple of inputs, cached prefixes cost a fraction, memory scales with context, and batch is cheap. Those relationships come from the hardware constraints described above, and they have held across every price cut so far. A design that exploits them stays efficient as the absolute numbers move.

Put the budget in the spec

Most product specifications list latency targets and say nothing about tokens. A better spec states, for each user-facing task, the time to first token at the 95th percentile, the total time at the 95th percentile, and a token budget split into cached input, fresh input, and output. The budget produces a cost per task, and the cost per task, set against what a successful task is worth, tells you whether the product has a margin.

Designs that cannot state those numbers are not ready for review. A team that can state them has made choices about prefix layout, reply length, context limits, batching and routing, and will notice when a change in any of them moves the bill. The first change to try is the cheapest one on the list: find the longest static prefix in your largest workload, move it to the front of the prompt, and turn on caching.

Sources

1. Meta (2024). Llama 3.1 70B model configuration (80 layers, 8 key-value heads, head dimension 128, 131,072 positions). Hugging Face model repository.
2. Kwon, W., Li, Z., Zhuang, S., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180.
3. NVIDIA. H100 Tensor Core GPU datasheet (memory bandwidth and BF16 throughput).
4. Anthropic. Pricing, Claude developer documentation (list prices as of October 2026). platform.claude.com/docs/en/about-claude/pricing.
5. OpenAI (August 2025). GPT-5 API pricing announcement.
6. Epoch AI. LLM inference price trends. epoch.ai/data-insights/llm-inference-price-trends.
7. Stanford Institute for Human-Centered AI (2025). The AI Index Report 2025.
8. Anthropic (June 2025). How we built our multi-agent research system. anthropic.com/engineering/multi-agent-research-system.
9. Ainslie, J., Lee-Thorp, J., de Jong, M., et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023. arXiv:2305.13245.
10. Leviathan, Y., Kalman, M., Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023. arXiv:2211.17192.