Throughput and Little's law

Inference and Serving, lesson 8

Requests per second, requests in flight, and the time each one takes.

You know TTFT and tokens per second for one request ("First token and every token after"), and how much cache memory a request holds ("How big the cache gets"). This lesson is about a server with many users at once.

Part 1. One at a time

Picture a server that answers one request, then the next. Each answer is 200 tokens at 25 ms per token, so each takes about 5 seconds. In one minute it finishes 12 requests.

That's 0.2 requests per second, or 40 tokens per second across everyone. If 50 people show up in the same minute, most of them wait in line.

Throughput

How much work the whole server finishes per second, counted across all users. Usually given as requests per second or output tokens per second. TPS is one user's speed. Throughput is everyone's added together.

Your turn

A server handles one request at a time, and each takes 4 seconds. What is its throughput?

One request every 4 seconds is 1 ÷ 4 = 0.25 requests per second. Throughput and latency are linked this simply only when nothing overlaps. Once a server works on several requests at once, the two come apart.

Show the answer

D. One request every 4 seconds is 1 ÷ 4 = 0.25 requests per second. Throughput and latency are linked this simply only when nothing overlaps. Once a server works on several requests at once, the two come apart.

Part 2. Many at once

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Many at once
Requests in flight
Sizing a server with it

See Pro plans I have Pro: sign in

Next lessonGoodput under errors and timeoutsCount only the requests that succeed within their deadline. Start lesson 9

Inference and Serving overviewSpotted a mistake? Tell us.

08 · Throughput and Little's law