insideout Get Pro

10 · Time until the answer is usable

Pro

Inference and Serving, lesson 10

Time until the answer is usable

End-to-end time from request to a result someone can act on.

You know TTFT and the gaps between tokens ("First token and every token after"), and why a target has to hold for every request ("Goodput under errors and timeouts").

Part 1. When streaming can't help

Your code asks a model for a product listing as JSON: title, price, ten bullet points. The model starts streaming after half a second. Your code can't do anything with it yet.

Half a JSON object won't parse. A tool call is useless until its arguments are complete. A summary for a newsletter can't go out until it ends. In all of these, nobody reads along: the result becomes usable only when the last token has arrived.

Your turn

The JSON reply is 400 tokens, TTFT is 0.5 s and each token after the first takes 20 ms. When can your code use the result?

0.5 s for the first token, then 399 more at 20 ms each: 0.5 + 7.98 = 8.48 s. Streaming makes a human reader feel it started at 0.5 s. A program that needs the whole object sees only the 8.48.

Part 2. From request to usable answer

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. From request to usable answer
  2. What actually moves it
  3. Using the answer before it's finished
See Pro plansI have Pro: sign in
Next lessonReading a roofline chartWhether a workload is limited by compute or by memory bandwidth. Start lesson 11

Inference and Serving overviewSpotted a mistake? Tell us.