First token and every token after

Inference and Serving, lesson 5

Time to first token, time per token, and how streaming changes the wait.

You know the two phases of a request from "Reading the prompt, writing the reply". If you've watched a chatbot's reply appear word by word, you've already seen both.

Part 1. Two waits on a stopwatch

Send a question to a chatbot and watch closely. First there's a pause where nothing happens. Then words start to appear, and they keep coming at a steady pace until the answer is done.

Those are two different waits, and they feel different. The pause decides whether the app feels alive. The pace decides whether reading the answer feels smooth or like watching paint dry.

Example 1.1

One reply, on a stopwatch

0.00 syou press send
The clock starts the moment the request leaves your machine.
0.00 s→0.60 sfirst word appears
0.6 seconds of blank screen. This is the first wait.
0.600 s0.625 s0.650 s0.675 s...
After that, a new token arrives every 25 milliseconds. That's 40 tokens each second.
200 tokens→5.575 slast token
The reply is 200 tokens. The first came at 0.6 s and the other 199 took 25 ms each: 0.6 + 199 × 0.025 = 5.575 s.

Time to first token (TTFT)

The blank-screen wait. The clock starts when the request leaves the client and stops when the reply's first token shows up there. In the stopwatch above it's 0.6 seconds.

Tokens per second (TPS)

How fast the reply comes out once it has started: the tokens after the first, divided by the time they took. Above, 199 tokens in 4.975 seconds is 40 TPS. Some tools quote the same thing the other way up, as milliseconds per token (25 ms here).

Your turn

A reply has a TTFT of 0.3 s and then streams at 8 TPS. Another has a TTFT of 3 s and then streams at 80 TPS. Both write 40 tokens. Which one finishes first?

First: 0.3 + 39 ÷ 8 ≈ 5.2 s. Second: 3 + 39 ÷ 80 ≈ 3.5 s. The second finishes about 1.7 seconds earlier, even though it sits blank for ten times as long. Which one feels better depends on the reader, and that's why you track both numbers.

Show the answer

C. First: 0.3 + 39 ÷ 8 ≈ 5.2 s. Second: 3 + 39 ÷ 80 ≈ 3.5 s. The second finishes about 1.7 seconds earlier, even though it sits blank for ten times as long. Which one feels better depends on the reader, and that's why you track both numbers.

Part 2. What fills the first wait

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

What fills the first wait
Streaming moves the wait
The gaps between tokens
Measuring it yourself

See Pro plans I have Pro: sign in

Next lessonPercentiles and the slow tailWhy p99 matters more than the average once traffic is real. Start lesson 6

Inference and Serving overviewSpotted a mistake? Tell us.

05 · First token and every token after