insideout Get Pro

09 · Goodput under errors and timeouts

Pro

Inference and Serving, lesson 9

Goodput under errors and timeouts

Count only the requests that succeed within their deadline.

You know throughput, work per second across all users ("Throughput and Little's law"), and percentiles ("Percentiles and the slow tail").

Part 1. Throughput counts the wrong things

A load test reports 100 requests per second. Looks great. Then you read the details: 8 of every 100 came back as errors, and another 12 took so long that the user had already closed the tab.

Your turn

Of those 100 requests per second, how many did a user actually get value from?

Take away the 8 errors and the 12 that arrived after the user gave up. 100 − 8 − 12 = 80. The server spent GPU time on all 100, but only 80 counted.

Throughput counts everything the server finished, useful or not. A slow answer still counts. A timeout that got retried can count twice. You want a number that only counts the answers that did their job.

Goodput

Requests per second that succeeded and met every latency target. Some dashboards give it as a share instead: 80% of requests were good. Multiply the share by the request rate to get the rate.

Part 2. Writing down "good"

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Writing down "good"
  2. Push harder, get less
  3. Finding the rate to run at
See Pro plansI have Pro: sign in
Next lessonTime until the answer is usableEnd-to-end time from request to a result someone can act on. Start lesson 10

Inference and Serving overviewSpotted a mistake? Tell us.