Capstone: profile and fix a slow model

Inference and Serving, lesson 25

Find the slowest part of a serving setup and fix it.

This walkthrough uses ideas from across the module: "Percentiles and the slow tail", "Continuous batching", "Reading a roofline chart", "Prefix caching", "Number formats" and "Pricing a single request". Nothing new is introduced.

Part 1. Measure before touching anything

A team runs a support chat on its own GPUs: an 8B model with Llama 3 8B's shape (32 layers, 8 key-value heads of 128 dims), in BF16, so 16 GB of weights. Users say it's slow to start answering. Finance says the five GPUs it needs at peak cost too much. Every price and measurement below is an example, but each one follows from the ideas in this module.

Before changing anything, the team writes down targets and a fair benchmark. Targets: p95 TTFT under 1 second, at least 30 tokens per second per user, at the peak rate of 6 requests per second. Benchmark: a fixed sample of real prompts (about 4,000 tokens in, 250 out), warm-up requests discarded, open-loop traffic at 6 per second, five runs per config.

Example 1.1

The baseline

p95 TTFT2.8 s
per user98 tokens/s
Streaming speed is fine. Starting is not.
prefill of 4,000 tokens0.4 s
Server logs show prefill runs at about 10,000 tokens per second, so a prompt takes 0.4 s.
GPU time per request0.72 s
× 6 per second=4.3 GPUs at peak
Each request costs 0.72 GPU-seconds. At 6 requests per second that's 4.3 GPUs of work, so they run 5. At an example $2 an hour, that's $7,300 a month.

Your turn

p95 TTFT is 2.8 s, but prefill only takes 0.4 s. Where do the other 2.4 s go?

TTFT is waiting plus prefill plus one decode step. Tokenizing and one step take milliseconds. The logs show requests sitting in a queue: the server uses static batching with a batch of 8, so a new request waits for the current batch to finish all 250 of its tokens. That takes 250 × 10.2 ms ≈ 2.5 s.

Show the answer

C. TTFT is waiting plus prefill plus one decode step. Tokenizing and one step take milliseconds. The logs show requests sitting in a queue: the server uses static batching with a batch of 8, so a new request waits for the current batch to finish all 250 of its tokens. That takes 250 × 10.2 ms ≈ 2.5 s.

Part 2. First change: the queue

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

First change: the queue
Second change: stop recomputing the prompt
Third change: fewer bytes per step

See Pro plans I have Pro: sign in

The last lesson in Inference and ServingBack to the overviewWhere the time and money go Back to Inference and Serving

Inference and Serving overviewSpotted a mistake? Tell us.

25 · Capstone: profile and fix a slow model