insideout Get Pro

07 · Prompt length on different GPUs

Pro

Inference and Serving, lesson 7

Prompt length on different GPUs

How prompt length and hardware set prefill time.

You know the model reads the whole prompt (prefill) before it writes the first token ("Reading the prompt, writing the reply"), and that this sits inside TTFT ("First token and every token after").

Part 1. Longer prompt, longer wait

Paste a whole contract into a chatbot and the first word takes noticeably longer than for "hi". The model has to read every token of the prompt before it can write anything, and reading isn't free.

There's a handy rule for how much work that is. Each token passing through the model costs about 2 operations per parameter, one multiply and one add. For an 8-billion-parameter model that's 2 × 8 billion = 16 billion operations per prompt token.

Example 1.1

Prefill work for an 8B model

  1. 1,000 tokens×16 billion=16 trillion ops

    A short document. 16 trillion operations.

  2. 8,000 tokens×16 billion=128 trillion ops

    Eight times the prompt, eight times the work.

  3. 32,000 tokens×16 billion=512 trillion ops
    plus extra for attention

    At long prompts the attention part grows faster than this, because every token looks back at every token before it ("The price of a long context"). The rule of thumb starts to undercount.

FLOP

One floating-point operation: a single multiply or add on decimal numbers. Work is counted in FLOPs. Hardware speed is quoted in FLOP per second, usually written FLOPS, and modern GPUs reach hundreds of trillions (teraFLOPS, TFLOPS).

Your turn

A 4,000-token prompt has 0.13 s of prefill. You add retrieved documents and the prompt grows to 8,000 tokens. Roughly what happens to the prefill time?

Prefill work is about 2 × parameters × tokens, so twice the tokens is about twice the work. Attention's extra cost does grow with the square of the length, but at 8,000 tokens on an 8B model it's still a small part. It starts to matter much further out.

Part 2. Prefill keeps the GPU busy

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Prefill keeps the GPU busy
  2. Hardware sets the speed limit
  3. A number you can carry around
See Pro plansI have Pro: sign in
Next lessonThroughput and Little's lawRequests per second, requests in flight, and the time each one takes. Start lesson 8

Inference and Serving overviewSpotted a mistake? Tell us.