Designing a fair benchmark

Inference and Serving, lesson 21

Warm-up, fixed inputs, enough runs, and one change at a time.

You know TTFT, percentiles and goodput from "First token and every token after", "Percentiles and the slow tail" and "Goodput under errors and timeouts".

Part 1. The first requests lie

You start a fresh server and send 10 requests one after another. TTFT in milliseconds: 2,400, 610, 190, 185, 192, 188, 183, 191, 187, 186.

Your turn

What's the mean TTFT of those 10 requests, and is it a fair number for this server?

The mean is 451 ms, but 8 of the 10 requests took under 192 ms. The first two paid one-off costs. Drop them and the mean is about 188 ms. A number that mixes start-up with steady state describes neither.

Show the answer

B. The mean is 451 ms, but 8 of the 10 requests took under 192 ms. The first two paid one-off costs. Drop them and the mean is about 188 ms. A number that mixes start-up with steady state describes neither.

The first requests to a fresh server pay for things that happen once: loading weights into GPU memory, compiling and tuning kernels for the shapes it sees, capturing CUDA graphs, filling the prefix cache. Real users mostly meet a server that's been running for a while.

Warm-up

Requests you send and then throw away before measuring, so one-off start-up costs don't land in your numbers. Send enough to cover the prompt lengths and batch sizes you'll test. Measure cold start separately, if it matters to you, and label it.

Part 2. One run proves little

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

One run proves little
Change one thing
Send traffic the way users do

See Pro plans I have Pro: sign in

Next lessonServing a mixture of expertsEvery expert sits in memory, but each token reads only a few of them. Start lesson 22

Inference and Serving overviewSpotted a mistake? Tell us.

21 · Designing a fair benchmark