insideout Get Pro

20 · Pricing a single request

Pro

Inference and Serving, lesson 20

Pricing a single request

Turn tokens, hardware and utilisation into a cost per request.

You know throughput and goodput ("Throughput and Little's law", "Goodput under errors and timeouts"), and how batch size changes tokens per second ("Reading a roofline chart").

Part 1. Tokens in, tokens out, dollars

A hosted API bills tokens, and it bills input and output at different rates. Take example prices: $3 per million input tokens and $15 per million output tokens. Real prices vary by provider and model and change often, so treat these as stand-ins.

One chat request sends 2,000 tokens (system prompt, history, question) and gets back 400.

Your turn

Output costs 5 times as much per token. With 2,000 tokens in and 400 out, which side costs more?

2,000 × $3 per million is $0.006. 400 × $15 per million is also $0.006. Five times the tokens at a fifth of the price. Long prompts with short answers are input-heavy; agents that write a lot are output-heavy.

Example 1.1

One request, then a month of them

  1. 2,000 in×$3 per M=$0.006

    Input tokens at the input price.

  2. 400 out×$15 per M=$0.006

    Output tokens at the output price.

  3. $0.006+$0.006=$0.012

    About a cent per request.

  4. 50,000 a day×30 days×$0.012=$18,000

    A month at 50,000 requests a day. Small per-request numbers add up fast.

Output usually costs more because of how it's made. Prefill handles all the input tokens in one parallel pass. Decode makes output one memory-bound step at a time, as in "Reading a roofline chart", so each output token holds the GPU for longer.

Part 2. Self-hosting: dollars per GPU hour

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Self-hosting: dollars per GPU hour
  2. Batch size sets the price of a token
  3. Cost per request on your own GPUs
  4. Hidden multipliers
See Pro plansI have Pro: sign in
Next lessonDesigning a fair benchmarkWarm-up, fixed inputs, enough runs, and one change at a time. Start lesson 21

Inference and Serving overviewSpotted a mistake? Tell us.