Inference and Serving, lesson 24
Running a model server
Read what the engine reports, and add servers on the waiting queue.
Once a model serves real traffic, you see it through the numbers its engine reports, and you add or remove servers by them. This lesson reads those numbers through a traffic spike, then builds a scaling rule from them. You need Little's law from "Throughput and Little's law" and preemption from "Continuous batching".
By the end, you can:
- Turn two readings of a server's metrics into rates, times and averages.
- Read a dashboard during a spike, and tell a full server from a slow one.
- Spot preemption on a dashboard, price the repeated work, and size a cap that stops it.
- Set a target for requests waiting from Little's law, and see why CPU misses a GPU server's load.
Part 1. Gauges and counters
Turn two readings of a server's metrics into rates, times and averages.
A serving engine such as vLLM publishes its numbers on a /metrics page, and a monitoring system such as Prometheus reads it at regular intervals. Some numbers are readings of right now: requests running, requests waiting, the share of the KV cache in use. Others only ever count up: requests finished, tokens written, preemptions.
Example 1.1
Two readings, one minute apart
A Llama 3.1 8B server is read at 14:00 and at 14:01. Requests running reads 48 both times. Requests finished goes from 12,040 to 12,400 and tokens written from 3,612,000 to 3,720,000. The TTFT histogram's sum of seconds goes from 2,516 to 2,696, and its count from 12,040 to 12,400. How busy is it?
- Requests a second(12,400 − 12,040) ÷ 60 = 6
A counter's change divided by the seconds between readings.
- Tokens a second108,000 ÷ 60 = 1,800
That's 300 tokens per finished request.
- Time running48 ÷ 6 = 8 s
Little's law turned around: requests inside ÷ requests a second = time each one spends inside.
- Mean TTFT180 s ÷ 360 = 0.5 s
The histogram's sum and count are counters too. Divide their changes for the mean over the minute.
Answer6 requests and 1,800 tokens a second, each request running 8 s, with a mean TTFT of 0.5 s.
A counter's value covers everything since the server started, so only its change over a window says anything about now. A gauge you can read straight off, but it jumps from step to step, so chart it rather than trust one reading.
Gauge and counter
A gauge is a reading of right now that can go up or down, like requests waiting. A counter only goes up, like tokens written, and what you use is its rate. A latency histogram is a set of counters: one per bucket, plus a sum and a count.
rate = (later − earlier) ÷ seconds between readings
Counters drop to zero when a server restarts, and a plain subtraction then goes negative. Prometheus's rate functions allow for that. Names differ between engines and versions: vLLM's gauges are vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc, so check the list for your version.
Your turn
Two readings 120 s apart: requests finished goes from 5,000 to 5,600, and requests running reads 60 both times. About how long does each request spend running?
Part 2. Reading a spike
Read a dashboard during a spike, and tell a full server from a slow one.
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Reading a spike
- When the cache runs short
- Scaling on the queue
Inference and Serving overviewSpotted a mistake? Tell us.