Mixture of experts splits the price of a parameter in two

Mistral's Mixtral model, released in January 2024, has eight feed-forward blocks in every layer and uses two of them for each token. Its parameter count comes in two figures: 47 billion in total, 13 billion active during inference. On the benchmarks in its paper it matched or beat Llama 2 70B and GPT-3.5, with especially large gains in mathematics, code generation and multilingual tasks. Eleven months later DeepSeek-V3 pushed the ratio much further. It has 671 billion parameters, activates 37 billion for each token, was trained on 14.8 trillion tokens, and used 2.788 million H800 GPU-hours to do it.

Both models carry two numbers in their specifications, and the gap between them is the subject of this essay. In a dense model, one number covers everything: the parameter count determines how much arithmetic each token costs and how much memory the model occupies. A mixture-of-experts model separates the two. Arithmetic follows the active parameters. Memory follows the total. Most of what is surprising about the economics of these models, both the cheap training runs and the sometimes disappointing serving costs, comes from that separation.

An old idea that needed hardware to catch up

The idea dates to Noam Shazeer and colleagues in 2017, including Geoffrey Hinton and Jeff Dean. They inserted a layer containing up to thousands of small feed-forward networks, the experts, between the layers of an LSTM, and trained a gating network to choose a sparse combination of them for each input. The largest model had 137 billion parameters, and they reported more than a thousandfold increase in capacity with only minor losses in computational efficiency. In 2021 Switch Transformers, by William Fedus, Barret Zoph and Shazeer, simplified the routing so that each token goes to a single expert, trained models of up to a trillion parameters, and reported a fourfold speedup over T5-XXL.

For several years the approach stayed mostly inside large labs. It was difficult to train stably, and difficult to serve, for reasons that will become clear. By 2024 the engineering had matured enough that MoE designs became common among the largest open-weight releases. The pattern is the same everywhere: in each Transformer block, the attention layer stays as it is, and the single feed-forward network is replaced by a bank of experts and a small router that picks a few of them per token.

Arithmetic follows the active parameters

A dense transformer spends about two floating-point operations per parameter on each token it processes. A dense 70-billion-parameter model therefore costs about 140 gigaflops per token. Mixtral spends about 26 gigaflops, since 13 billion parameters are active. DeepSeek-V3 spends about 74 gigaflops per token, where a dense model of 671 billion parameters would spend 1.34 teraflops. That is a factor of 18 in compute for the same total capacity.

Training follows the same pattern. Using the usual approximation of six operations per parameter per token, DeepSeek-V3's run on 14.8 trillion tokens with 37 billion active parameters comes to about 3.3 × 10²⁴ operations. A dense model with 671 billion parameters on the same data would need about 6.0 × 10²⁵, eighteen times more. The report gives its own accounting of cost: 2.788 million GPU-hours at an assumed rental price of $2 each, or $5.576 million, a figure that covers the official training run and, as the report says, excludes the costs of prior research and ablation experiments. Those caveats matter, since the experiments that led to a design are a real part of its cost, but the arithmetic above shows why a run of that size was affordable at all.

A cross-check against a dense model supports the arithmetic. Meta's model card for Llama 3.1 405B, a dense model trained on about 15 trillion tokens, lists 30.84 million H100 GPU-hours. DeepSeek-V3's 2.788 million H800 GPU-hours is about eleven times fewer. The six-operations rule predicts the same ratio from the model sizes alone: 6 × 405 billion × 15 trillion is about 3.6 × 10²⁵ operations, against 3.3 × 10²⁴ for DeepSeek-V3, a factor of 11.1. The two chips deliver similar arithmetic throughput, so the saving in GPU-hours is the saving in active parameters, nearly to the decimal.

Beyond cost, the design brings a quality argument. The Mixtral result suggests that spreading knowledge across many specialized blocks and consulting only a few of them can match a dense model with five times the active parameters. Despite the name, the experts are not experts in a human sense. Mixtral's authors analyzed their router and found no obvious assignment of experts by topic, with some regularities that follow syntax. Calling them specialists is a metaphor, and the benefit is better described as capacity that costs nothing until it is used.

More, smaller experts

Mixtral uses 8 experts per layer and picks 2, which allows 28 distinct combinations for each token at each layer. DeepSeek-V3 splits the same feed-forward capacity into 256 routed experts and picks 8, which allows about 4.1 × 10¹⁴ combinations, plus a shared expert that every token passes through. The reasoning behind the finer grain is that a router with more and smaller choices can compose a more specific blend for each token, while the shared expert holds the knowledge that nearly every token needs, which would otherwise be copied into many of the routed ones. Whether the larger number of combinations is used well is an empirical question, and the model's results suggest that it is. The finer grain does have a price in serving, because it spreads a token's work across more destinations, as the next two sections show.

The weights still have to live somewhere

Every expert must be available at every step, because the router decides which ones a token needs only after the token arrives. All 671 billion parameters of DeepSeek-V3 must therefore be resident in accelerator memory. At 8-bit precision that is 671 GB, which does not fit in the 640 GB of a node with eight 80-GB GPUs. It does fit in a node of eight 141-GB cards (1,128 GB in all), with about 450 GB left for caches and activations. A dense model with 37 billion parameters would fit on one GPU at 8 bits, with room to spare.

This is the first place a buyer meets the split. A model sold as "37B active" is priced and benchmarked like a mid-sized model, and hosted like a very large one. For anyone running open-weight models on their own hardware, whether for data-sovereignty reasons or to control cost at high volume, the hardware has to be sized by the total parameter count. The compute budget alone leads to a node that cannot hold the model.

Decoding shrinks the saving

The second place is subtler. Recall that during generation the hardware is mostly waiting for memory: each step must read the weights that participate in that step. For a dense model, those are all the weights, once per step, however many conversations share the step. For a mixture of experts, a step reads the experts that at least one token in the batch has been routed to.

A small calculation shows how fast that set grows. Suppose each token picks its experts uniformly at random, which exaggerates the spread, since real routers have favorites. Mixtral picks 2 of 8 experts per layer. The chance that a given expert is untouched by a batch of B tokens is 0.75^B, so the share of experts that are read is 1 − 0.75^B. At B = 4 that is 68%, at B = 8 it is 90%, and at B = 16 it is 99%. By the time sixteen conversations share a step, the server reads essentially all 47 billion parameters per step instead of 13 billion, and the bandwidth cost of decoding looks like that of a dense 47-billion-parameter model. A dense 13-billion-parameter model serving the same sixteen conversations would read 3.6 times fewer bytes.

DeepSeek-V3 picks 8 of 256 routed experts per layer, plus one shared expert that every token uses. A routed expert is untouched with probability (1 − 8/256)^B. At B = 32 about 64% of the experts are read, at B = 128 about 98%, and at B = 256 essentially all. The saving that looked like a factor of 18 on paper exists in the memory-bound regime only when the batch is small enough that most experts sit idle, and small batches are exactly where the hardware is least efficient.

The two phases of serving behave differently here. Prefill processes a whole prompt in one pass, with thousands of tokens flowing through each layer, so every expert receives plenty of work and the step is limited by arithmetic, not memory. There the sparsity pays in full, and a long prompt costs what the active parameters say it should. Decoding is where the memory bill reappears. A workload dominated by long prompts and short answers, such as classifying or summarizing documents, therefore suits these models better than one dominated by long generations to a handful of users.

The way out is to make each GPU hold only a few experts and to send each token to wherever its experts live. This is called expert parallelism, and it turns the problem into one of very large batches spread across many GPUs, so that each GPU reads only its own few experts and still sees plenty of tokens. DeepSeek's report describes the scale this takes. Its prefill stage uses a minimum deployment unit of 4 nodes and 32 GPUs, and its decoding stage uses 40 nodes and 320 GPUs, with each GPU hosting one expert and some hosting redundant copies of popular ones to balance the load. That is a serving setup for a provider with enough traffic to fill hundreds of GPUs. It is not a configuration a company with twenty concurrent users can reproduce, and for such a company the cheap-per-token promise does not transfer.

What this means for choosing and pricing

Several practical rules follow.

First, read the two numbers separately and compute your own costs from them. Active parameters govern compute per token and therefore the training bill and the speed of a large batch. Total parameters govern memory, and therefore the minimum hardware.

Second, benchmark at your concurrency. Throughput numbers published for MoE models are often measured at large batch sizes on large clusters. A sensible test is to run your own load, at your own expected number of simultaneous requests, and measure latency and tokens per second. A model that looks four times cheaper at batch 256 may look no cheaper than a dense alternative at batch 8.

Third, understand why hosted MoE models can be priced low. A provider with large and steady traffic can fill the batches that make expert parallelism efficient. The price you pay reflects its scale as well as the architecture. That also means that a hosted MoE model and a self-hosted one are not the same economic object, even when the weights are identical.

Fourth, note the training-side risks, which show up as differences in behavior under load. Routers tend to collapse onto a few favorite experts unless something pushes them to spread the load. The usual fix is an auxiliary loss that penalizes imbalance, which can also degrade quality, and DeepSeek-V3's report describes an alternative it calls auxiliary-loss-free load balancing, which it presents as a first. Whether a given model balances well affects how it behaves under real traffic.

When a dense model is still the better answer

The sparse design is not a universal improvement. If a workload runs at low concurrency, say an internal assistant used by a few dozen people, a dense model with the same active size can be cheaper to host and simpler to operate, because it needs less memory and has no routing variance. If memory is the constraint, as on edge devices or modest on-premises servers, active parameters are the wrong yardstick and total parameters are the right one.

The reverse also holds. If a provider's traffic is large and steady, a mixture of experts lets it offer quality that would otherwise cost much more, and it is the design behind several of the largest open-weight models. The reasonable response from a buyer is to stop treating a headline parameter count as a measure of cost, and to ask for the two numbers, the concurrency at which a throughput figure was measured, and the hardware that figure assumed.

The first step before any purchase is a table with one row per candidate model and four columns: total parameters, active parameters, weight memory at the precision you will use, and measured tokens per second at your own concurrency. The table takes a day to fill in, and it will rank the candidates differently from the marketing numbers.

Sources

1. Jiang, A. Q., Sablayrolles, A., Roux, A., et al. (2024). Mixtral of Experts. Mistral AI. arXiv:2401.04088.
2. DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437.
3. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. arXiv:1701.06538.
4. Fedus, W., Zoph, B., Shazeer, N. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research 23. arXiv:2101.03961.
5. NVIDIA. H100 and H200 Tensor Core GPU datasheets (memory capacity: 80 GB and 141 GB).
6. Meta (2024). Llama 3.1 model card, training time in GPU hours. github.com/meta-llama/llama-models.