insideout Get Pro

23 · Contexts that don't fit

Pro

Inference and Serving, lesson 23

Contexts that don't fit

Choose which keys and values to keep when a chat outgrows memory.

A long chat, or an agent that runs all day, adds keys and values with every token until its cache outgrows its memory. Then something has to go. This lesson works out what each choice of what to keep saves, and what it costs. You need the cache per token from "How big the cache gets" and the window from "The price of a long context".

By the end, you can:

  1. Size a cache that keeps only the most recent tokens.
  2. Explain why a window needs the first tokens, and what keeping them restores.
  3. Pick which entry to evict by the attention each one has received.
  4. Compare what each rule keeps at the same memory, and what it costs in answers.

Part 1. Keep the last W tokens

Size a cache that keeps only the most recent tokens.

The first Mistral 7B has the cache shape of Llama 3 8B: 32 layers, 8 KV heads, head size 128, 2 bytes a number, so 2 × 8 × 128 × 32 × 2 = 131,072 bytes per token. But each of Mistral's layers was trained to look back only 4,096 tokens. Older entries are never read again.

Example 1.1

A 32,000-token conversation on Mistral 7B

Mistral 7B keeps 131,072 bytes of cache per token. Say 60 GB of GPU memory is free for caches. How big is a 32,000-token conversation's cache if you keep every token, and if you keep only the last 4,096? How many conversations fit each way?

  1. Keep every token32,000 × 131,072 B = 4.19 GB

    The cache grows by 128 KiB with every token.

  2. Keep the last 4,0964,096 × 131,072 B = 0.54 GB

    Past 4,096 tokens, each new token's entry replaces the oldest one.

  3. Compare32,000 ÷ 4,096 = 7.8

    Mistral's paper reports 8 times less cache at this length, with no loss in quality.

  4. Count what fits60 ÷ 4.19 = 14, 60 ÷ 0.54 = 111

    Rounded down to whole conversations.

Answer4.19 GB against 0.54 GB, so 14 conversations fit, or 111.

Nothing is lost, because the model never reads past its window. Older information still reaches it, moving forward up to 4,096 tokens per layer, but inside newer tokens' vectors, not through old cache entries.

The text so far, tokens 0 to 11012345678910110 to 3 are overwrittenThe cache: W = 8 slots8slot 09slot 110slot 211slot 34slot 45slot 56slot 67slot 7token 11 → slot 11 mod 8 = 3, over token 3
The text so far, tokens 0 to 11012345678910110 to 3 are overwrittenThe cache: W = 8 slots809110211344556677token 11 → slot 3, over token 3
Figure 1A rolling buffer with 8 slots. Each token's keys and values go in slot i mod 8, over the token 8 places back. Mistral's buffer works the same way with 4,096 slots.

Rolling buffer cache

A cache of a fixed W slots. The keys and values of the token at position i go in slot i mod W (the remainder after dividing i by W), over the token W places back. The cache stops growing once the text passes W tokens.

slot = i mod W, cache = W × bytes per token

The trap is a model trained to see every earlier token. Llama 3.1 8B keeps the same 128 KiB per token, and its cache could be cut the same way. But it never learned to work without its first tokens, as the next part shows.

Your turn

The Mistral 7B conversation reaches 100,000 tokens. How big is its cache with the 4,096-token rolling buffer, and without one?

Without a buffer every token keeps its entry: 100,000 × 131,072 B = 13.1 GB. The buffer holds 4,096 entries however long the text runs: 0.54 GB, 24 times less. 4.19 GB is the 32,000-token cache.

Part 2. Keep the first four as well

Explain why a window needs the first tokens, and what keeping them restores.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Keep the first four as well
  2. Keep the tokens that get attention
  3. What each rule forgets
See Pro plansI have Pro: sign in
Next lessonRunning a model serverRead what the engine reports, and add servers on the waiting queue. Start lesson 24

Inference and Serving overviewSpotted a mistake? Tell us.