insideout Get Pro

13 · Continuous batching

Pro

Inference and Serving, lesson 13

Continuous batching

Static, dynamic and continuous batching, and what each does to latency.

You know why a batch shares the cost of reading the weights ("Throughput and Little's law", "Arithmetic intensity"), and how a paged cache hands out memory in blocks ("How big the cache gets").

Part 1. Fill up, run to the end

The simplest server waits for four requests, starts them together, and runs until every one is done. Then it takes the next four. Say these four need 3, 10, 4 and 6 output tokens.

Example 1.1

Four slots, tokens left in each

  1. step 0
    31046

    All four start together. Each step makes one token in every busy slot.

  2. after step 3
    done713

    The first request is finished. Its slot sits empty, even if people are waiting.

  3. after step 6
    done4donedone

    Three slots idle. The GPU still reads all the weights each step, to make one token.

  4. after step 10
    donedonedonedone
    23 tokens ÷ 40 slot-steps=57.5% busy

    The longest request decides when the batch ends. Only 23 of 40 slot-steps made a token.

Static batching

Form a batch, run it until every request in it is done, then form the next. Easy to build. Wasteful when output lengths vary, which for chat they always do.

Your turn

With static batching, a new request arrives just after step 4 of that batch. When does it start?

The batch runs until its longest member (10 tokens) is done. The new request waits about 6 steps for nothing, and all of that lands in its TTFT.

Part 2. Wait a little, then go

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Wait a little, then go
  2. Refill every step
  3. What the scheduler does each step
  4. When the cache runs out
See Pro plansI have Pro: sign in
Next lessonNumber formatsWhat FP32, BF16, FP8, INT8 and INT4 each keep and lose. Start lesson 14

Inference and Serving overviewSpotted a mistake? Tell us.