insideout Get Pro

04 · The price of a long context

Pro

Inference and Serving, lesson 4

The price of a long context

What longer prompts cost in memory, compute and attention.

You need the attention scores from "Why the scores are scaled", and the cache size per token from "How big the cache gets". This lesson follows one long prompt through compute, memory and attention.

Part 1. Every token scores every token

In a 4-token text, each token computes a score against each token it can see. Ignore the mask for a moment and that's 4 × 4 = 16 scores. A 1,000-token text needs 1,000 × 1,000 = 1,000,000. And that's for one head in one layer.

Your turn

A head computes 1,000,000 scores for a 1,000-token text. How many does it compute for 2,000 tokens?

Both sides of the grid double: 2,000 × 2,000 = 4,000,000. Twice the text, four times the scores. The causal mask skips about half the grid, but half of a square still grows like a square.

Example 1.1

How the score grid grows

  1. 1,000 tokens→1 million scores

    A page or two of text.

  2. 2,000 tokens→4 million

    Double the length, four times the scores.

  3. 4,000 tokens→16 million

    Double again, four times again.

  4. 8,000 tokens→64 million

    Eight times the text of the first step costs 64 times the scores. Per head, per layer.

Quadratic cost, O(n²)

Work that grows with the square of the length n. Read O(n²) aloud as "order n squared": double n and the work goes up about four times. Work that grows in step with n is linear, O(n).

Part 2. Where the square starts to win

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Where the square starts to win
  2. The grid that doesn't fit
  3. Memory grows in a straight line
  4. Ways to pay less
See Pro plansI have Pro: sign in
Next lessonFirst token and every token afterTime to first token, time per token, and how streaming changes the wait. Start lesson 5

Inference and Serving overviewSpotted a mistake? Tell us.