Compare
Mixture of experts vs dense models
More weights in memory, less arithmetic for each token.
The short answer
A dense model runs every weight for every token. A mixture of experts (MoE) stores many MLPs in each block and runs only a few per token, so it does the arithmetic of a much smaller model: Mixtral 8x7B stores 46.7B weights and uses 12.9B per token. It still needs memory for every weight, so it pays off when your GPUs can hold the whole model and your traffic keeps the experts busy.
Mixture of experts
Use it when
- You want a large model's quality at a smaller model's compute per token.
- Your GPUs can hold every expert, with room left for the KV cache.
- You serve many requests at once, so batches keep the experts busy.
- What it costs
- Memory for all of it: Mixtral 8x7B needs about 93 GB for its weights in 16-bit, though each token uses 12.9B of its 46.7B weights. Training adds a router to keep balanced and, at scale, experts spread across GPUs.
- Watch out
- One request reads only its chosen experts, but a large batch reads nearly all of them at each step, so memory traffic grows towards that of a dense model of the full size.
A dense model
Use it when
- Memory is the limit, such as a single GPU or a laptop.
- Traffic is light, so batches stay small.
- You want the simplest model to fine-tune, quantize and serve.
- What it costs
- Every weight runs for every token, at about 2 operations per weight: a dense 70B model does about 140 billion operations per token.
- Watch out
- To match a large MoE's quality it needs more active weights, and every one of them costs arithmetic on every token.
At a glance
| Question | Mixture of experts | A dense model |
|---|---|---|
| Weights used per token | A few experts per block: Mixtral uses 12.9B of 46.7B | All of them |
| Memory for weights | Every expert, so set by the total | Every weight, all of which it uses |
| Compute per token | About 2 operations per active weight | About 2 operations per weight |
| Small batches | Reads only the chosen experts, so faster than a dense model of the same total | Reads every weight at each step |
| Large batches | Reads nearly every expert, but does less arithmetic per token | Reads every weight and does all the arithmetic |
| Training | A router to keep balanced; experts spread across GPUs at scale | No routing to manage |
| Reading the name | Qwen3-235B-A22B: 235B total, 22B active | Llama 2 70B: 70B, all active |
How to choose
- Write down your limit: GPU memory, delay per request, or cost per token at your real traffic.
- Check memory first: an MoE needs room for all its weights, plus the KV cache, on the GPUs you have.
- If it fits, compare it with the best dense model that fits the same GPUs, on your own evals.
- Measure speed and cost at the batch sizes you really serve, since an MoE's edge changes with batch size.
- If memory is tight, try a quantized MoE as well as a smaller dense model: gpt-oss-120b stores its experts in a 4-bit format to fit on one 80 GB GPU.
When you need both
Learn it properly
Questions
What does 8x7B mean in Mixtral 8x7B?
8 experts in each block of a model shaped like Mistral 7B. Only the MLPs come in eights, so the total is 46.7B weights, not 56B, and each token uses 2 experts: 12.9B weights.
What does A22B mean in Qwen3-235B-A22B?
A is for active: each token uses 22B of the model's 235B parameters. Each block has 128 experts, and a token uses 8 of them.
Does an MoE need less GPU memory than a dense model?
No. Every expert must be loaded, because any token may pick any of them, so memory follows the total. DeepSeek-V3 has 671B parameters to hold, though each token uses only 37B.
Is an MoE faster than a dense model?
Faster than a dense model of the same total size, yes: each token does the arithmetic of its active weights only. Against a dense model with the same active count, it needs several times the memory, and at large batches it reads far more of it at each step.
Why does an MoE need load balancing?
Left alone, the router keeps picking the experts that are already good, which then get better still, while the rest go unused. A small extra loss during training, or DeepSeek-V3's bias on each expert's score, spreads the tokens out. The mixture of experts lesson works one through by hand.