insideout Get Pro

22 · Serving a mixture of experts

Pro

Inference and Serving, lesson 22

Serving a mixture of experts

Every expert sits in memory, but each token reads only a few of them.

A mixture-of-experts model keeps several expert blocks in every layer and sends each token through only a few. For serving, memory pays for every expert, while each token's work pays only for its few. You need the router from "Mixture of experts" in Transformers, and memory-bound decode from "Reading the prompt, writing the reply".

By the end, you can:

  1. Work out an MoE model's memory, its work per token and the GPUs it needs.
  2. Work out how many experts a decode step reads as its batch grows.
  3. Count the bytes a token sends between GPUs, and see why the busiest GPU sets the pace.
  4. Size a model whose experts use fewer bits than the rest, and tell when offloading pays.

Part 1. Memory for every expert, work for a few

Work out an MoE model's memory, its work per token and the GPUs it needs.

Mixture of experts, in TransformersHow the router picks each token's experts, and what total and active parameters count.

Mixtral 8x7B has 8 experts in each of its 32 layers, and its router sends every token to 2 of them. Nobody knows in advance which 2, so all 8 have to sit in GPU memory, ready. Only the 2 that are picked do any work.

Example 1.1

Mixtral 8x7B against a dense 13B model

Mixtral has 46.7 billion weights, and each token uses 12.9 billion. In 16 bits, how much memory do the weights take, on how many 80 GB H100s, and how much work is a token? Compare a dense 13B model.

  1. Size every weight46.7 B × 2 = 93.4 GB

    Any token may pick any expert, so all of them stay loaded.

  2. Count the GPUs93.4 ÷ 80 = 1.17 → 2

    More than one H100 holds, before any KV cache.

  3. Count one token's work2 × 12.9 B = 25.8 G

    About 2 operations for each weight it uses.

  4. Size the dense 13B13 B × 2 = 26 GB

    One GPU with room to spare, and 26 billion operations a token.

Answer93.4 GB on 2 GPUs, for the work per token of a 13B model that fits on 1.

Memory holds every weight any token might need. Work counts only the weights a token uses. In a dense model those are the same weights. An MoE pulls them apart: Mixtral's experts hold 45.1 of its 46.7 billion weights, and each token uses a quarter of them.

Mixtral 8x7B stores every expertE1E2E3E4E5E6E7E846.7 B weights: 93.4 GB in 16 bitsOne token uses 2 of them, plus the restE1E2E3E4E5E6E7E812.9 B weights: 25.8 G operationsA dense 13B model stores and uses13 B weights: 26 GB, 26 G operationsattention and the rest: 1.6 Bone expert in all 32 layers: 5.6 B
Mixtral 8x7B stores every expertE1E2E3E4E5E6E7E846.7 B weights: 93.4 GB in 16 bitsOne token uses 2 of them, plus the restE1E2E3E4E5E6E7E812.9 B weights: 25.8 G operationsA dense 13B model stores and uses13 B weights: 26 GB, 26 G operationsattention and the rest: 1.6 Bone expert in all 32 layers: 5.6 B
Figure 1What Mixtral keeps in memory, and what one token uses. Memory pays for every box. Work pays only for the orange ones.

Total and active parameters

Total parameters count every weight a model stores, and set its memory. Active parameters count the weights one token uses, the shared parts plus k experts per layer, and set its work.

memory = total × bytes; work per token ≈ 2 × active

Read model names with care. 8 × 7 is 56, yet Mixtral holds 46.7 billion weights: only the feed-forward blocks come in 8 copies, and attention is shared. Names like Qwen3-235B-A22B give both numbers, 235 billion total and 22 billion active.

With one user, Mixtral does decode about as fast as a 13B model, since each step reads only the weights its token uses. Part 2 adds more users.

Your turn

DeepSeek-V3 ships its 671 billion weights in 8 bits, 1 byte each, and each token uses 37 billion. How many 80 GB GPUs just to hold the weights, and how much work is a token?

Memory follows the total: 671 GB ÷ 80 GB = 8.4, so 9 GPUs before any KV cache. Work follows the active weights: 2 × 37 billion = 74 billion operations. 1 GPU sizes memory from the active weights. 17 counts 2 bytes a weight.

Part 2. A batch reads more experts

Work out how many experts a decode step reads as its batch grows.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. A batch reads more experts
  2. Experts spread across GPUs
  3. Fewer bits for the experts
See Pro plansI have Pro: sign in
Next lessonContexts that don't fitChoose which keys and values to keep when a chat outgrows memory. Start lesson 23

Inference and Serving overviewSpotted a mistake? Tell us.