insideout Get Pro

03 · Spending limits per user and per team

Free

Production AI, lesson 3

Spending limits per user and per team

Token budgets, rate limits and quotas.

A model bill grows with whatever your code sends, so one bug can spend a month's budget overnight. This lesson sets limits at every level: how much each task, user and team may spend, and how fast. You need to know that calls are priced per token, as in "Routing requests between models".

By the end, you can:

  1. Work out what a runaway task costs, and what a budget caps it at.
  2. Run a token bucket by hand, and tell what its capacity and refill rate each control.
  3. Find which of a provider's limits runs out first for a given request size.
  4. Check a request against user, team and global limits, so one customer can't starve the rest.

Part 1. A budget at every level

Work out what a runaway task costs, and what a budget caps it at.

An agent gets stuck. It calls a tool, misreads the result and calls again, over and over, resending its whole history each time. Model bills don't grow with servers you can see. They grow with whatever your code decides to send.

Example 1.1

One stuck task, then a thousand

Each round resends 20,000 tokens of history, and the task loops 200 times. Input costs $2 per million tokens (Claude Sonnet 5.5, November 2026). Overnight, a bad deploy makes the same bug hit 1,000 users. What does the input cost, with and without a budget of 200,000 tokens per task?

  1. Count one task's tokens200 × 20,000 = 4,000,000

    Every round pays for the whole history again.

  2. Price one task4 × $2 = $8

    4 million tokens at $2 per million.

  3. Multiply by the users1,000 × $8 = $8,000

    All of it before anyone wakes up.

  4. Cap each task1,000 × $0.40 = $400

    200,000 tokens at $2 per million is $0.40, and the budget stops each task after its tenth round.

Answer$8,000 of input overnight without a budget, $400 with one. The replies add a little on top.

The budget doesn't make the bug cheaper to have. It makes the worst case small, and known in advance. That's why you want a limit at every level: per call, per task, per user per day and for the whole app per month, each catching what the others miss.

Token budget

A ceiling on the tokens, or the money, something may spend: one call, one task, one user per day, the whole app per month. A request that would go over is stopped, moved to a cheaper model or held until someone confirms.

worst case = how many of them × budget each

Per-user budgets make the worst day a sum you can do in advance: 20,000 free users at $0.20 a day each can cost at most $4,000. Make budgets soft before they're hard. At 80%, warn. At 100%, switch to a cheaper model or ask. Stop only after that.

Your turn

An agent resends 50,000 tokens a round, at $2 per million input tokens, under a budget of 500,000 tokens per task. When it gets stuck, how many rounds run, and what does the input cost?

500,000 ÷ 50,000 = 10 rounds before the budget stops it, and 500,000 tokens at $2 per million cost $1.00. $0.10 slips a decimal place, and 100 rounds would need a budget ten times bigger.

Part 2. Rate limits and the token bucket

Run a token bucket by hand, and tell what its capacity and refill rate each control.

Budgets cap the total. Rate limits cap the speed, so that one burst can't take everything at once. A common rate limiter works like a bucket that holds up to 10 tokens and refills at 2 tokens a second. Each request takes one out, and an empty bucket turns requests away.

These tokens are permits, not pieces of text. The two names just happen to collide.

Example 2.1

Capacity 10, refill 2 a second

The bucket starts full. 12 requests arrive at once at 0 s, 3 at 1 s, 1 at 2.5 s and 4 at 6 s. How many pass?

  1. A burst at 0 s10 pass, 2 refused

    The full bucket lets 10 through at once, and then it's empty.

  2. Refill, then 3 at 1 s0 + 1 × 2 = 2

    Two permits have dripped back in, so 2 pass and 1 is refused.

  3. Refill, then 1 at 2.5 s0 + 1.5 × 2 = 3

    It passes, leaving 2.

  4. Refill, then 4 at 6 s2 + 3.5 × 2 = 9

    All 4 pass, leaving 5. A quiet spell fills the bucket back up, but never past 10.

Answer17 of the 20 requests pass. The 3 refused all came in bursts.

capacity 100510refills 2 a second0 s10/121 s2/32.5 s1/16 s4/4
capacity 100510refills 2 a second0 s10/121 s2/32.5 s1/16 s4/4
Figure 1The bucket's level over 7 seconds. It drops at each burst and refills at 2 a second, never past 10. Under each burst: how many passed, of how many asked.

The two settings do different jobs. The capacity decides how big a burst gets through. The refill rate decides the long-run average: over a long stretch, nobody gets more than 2 requests a second, however they space them.

Token bucket

A rate limiter with two settings: a capacity, which sets the biggest burst, and a refill rate, which sets the long-run average. Anthropic's API uses one, so its limits refill continuously instead of resetting each minute.

level = min(capacity, level + seconds × refill rate)

So a per-minute limit can bite on a short burst. Anthropic's docs warn that 60 requests per minute may be enforced as 1 request per second: send 60 in the same second and some can be refused, though the minute's total is within the limit.

Your turn

Same bucket (capacity 10, refill 2 per second), full at the start. A client sends exactly 3 requests every second, forever. What happens?

The bucket drains by 1 each second (3 out, 2 in), so the full bucket covers the gap for about 8 seconds. Once it's empty, only the 2 refilled tokens per second are available, so 1 request in 3 is refused. Capacity buys time; the refill rate decides the long run.

Part 3. Count tokens, not just requests

Find which of a provider's limits runs out first for a given request size.

One request can be 50 tokens or 50,000. A limit of 100 requests a minute treats them the same, though one costs a thousand times the other. So model providers limit tokens as well as requests. Anthropic's API counts three things per minute: requests, input tokens and output tokens.

Example 3.1

Which limit binds first

On Anthropic's Start tier in November 2026, Claude Sonnet 5.5 allows 4,000 requests, 2,000,000 input tokens and 400,000 output tokens per minute. Each of your requests reads 10,000 tokens and writes 1,000. How many requests a minute can you send?

  1. Requests per minute4,000

    The request limit on its own.

  2. Input tokens per minute2,000,000 ÷ 10,000 = 200

    Only 200 requests' worth of input fit in a minute.

  3. Output tokens per minute400,000 ÷ 1,000 = 400

    The replies alone would allow 400.

  4. Take the smallest200

    Whichever limit runs out first throttles you.

Answer200 requests a minute. The input limit binds long before the request limit.

Each limit is its own bucket, and a request needs room in all of them. Long prompts, such as big retrieved documents or long chat histories, use up input limits fast. Trimming them lets more requests through, as well as lowering the bill.

RPM and TPM

Requests per minute and tokens per minute. Your account has both, often with input and output tokens counted apart. Whichever you run out of first is the one that throttles you.

requests a minute = the smallest of RPM and each TPM ÷ tokens per request

Providers count differently, so read your provider's docs. In November 2026, Anthropic doesn't count input tokens read from the prompt cache: if 8,000 of the 10,000 are cached, only 2,000 count, and the output limit binds at 400. OpenAI counts a request as at least its max_tokens, so set that close to the reply you expect.

Your turn

Your account allows 500 RPM and 100,000 TPM. Each request uses about 4,000 tokens. How many requests a minute can you actually send?

100,000 ÷ 4,000 = 25. The token limit bites long before the request limit. Long prompts (big retrieved documents, long chat histories) eat token limits fast, which is one reason to keep them trim.

Part 4. Limits per user, per team and overall

Check a request against user, team and global limits, so one customer can't starve the rest.

Say your app sells an AI writing tool to companies, each a team of users, all sharing your one provider account. One company starts a bulk job that rewrites its whole product catalogue. With no limits of your own, that job takes the whole token limit, and every other customer sees "try again later".

So every request passes three gates: the user's limit, the team's limit and the global one, which is what your provider account allows. It runs only if all three have room.

Example 4.1

Three gates in one minute

Per-minute limits are 20,000 tokens per user, 50,000 per team and 100,000 overall. Within one minute, one team sends: user 1, 15,000 tokens; user 1 again, 8,000; users 2 and 3, 15,000 each; user 4, 15,000. Which requests run?

  1. User 1's first request15,000 ≤ 20,000

    Every gate has room, so it runs and is charged at all three.

  2. User 1 again15,000 + 8,000 = 23,000

    Over the user's 20,000: refused, and nothing is charged.

  3. Users 2 and 315,000 + 30,000 = 45,000

    Both run, and the team reaches 45,000 of its 50,000.

  4. User 445,000 + 15,000 = 60,000

    Over the team's 50,000, so it's refused, though user 4 has used nothing.

Answer3 of the 5 requests run. One is stopped at the user gate and one at the team gate.

User 40 + 15k ≤ 20kTeam45k + 15k > 50kEveryone60k + 15k ≤ 100k
User 40 + 15k ≤ 20kTeam45k + 15k > 50kEveryone60k + 15k ≤ 100k
Figure 2User 4's request at the three gates. The user and global gates have room. The team gate doesn't, so the request is refused.

Each gate protects something different. The user gate stops one person using up their team's share. The team gate stops one customer using up everyone's. The global gate keeps you inside your provider's limits, so the provider never has to refuse you.

Noisy neighbour

A customer whose heavy use slows down or blocks everyone else on shared capacity. A cap per team contains it: the bulk job just runs slower.

admit only if every gate has room, then charge every gate

A cap is a ceiling, not a reservation. Anthropic's workspace limits work this way: cap one workspace at 30,000 input tokens a minute, and whatever it doesn't use stays available to the others. When you do refuse, send HTTP 429 with a Retry-After header, so clients back off by the right amount.

Your turn

Limits are 10,000 tokens a minute per user, 30,000 per team and 100,000 overall. This minute, a team has used 25,000 tokens, 3,000 of them by one user. That user now sends 6,000. What happens?

The user would reach 3,000 + 6,000 = 9,000, inside their 10,000. The team would reach 25,000 + 6,000 = 31,000, over its 30,000. One full gate is enough to refuse it.

Where you'll meet it

In interviews

"Design a rate limiter for an AI API" is a common system design question. Keep a token bucket per user and per team, charge by tokens rather than requests, hold the counts in a shared store such as Redis, and refuse with 429 and Retry-After. Then add budgets for the totals.

On the job

Every Anthropic reply carries headers such as anthropic-ratelimit-input-tokens-remaining, and OpenAI's carry x-ratelimit-remaining-tokens. Read them, and slow down before you're refused. Stripe's engineering blog describes the four limiters on its API, built on token buckets, including load shedders that drop low-priority traffic first when the system is strained.

A token bucket, in Python
class TokenBucket:
    def __init__(self, capacity, rate):
        self.capacity, self.rate = capacity, rate
        self.level, self.last = capacity, 0.0

    def take(self, now, cost=1):
        self.level = min(self.capacity, self.level + (now - self.last) * self.rate)
        self.last = now
        if self.level < cost:
            return False      # refused: answer 429 with Retry-After
        self.level -= cost
        return True

bucket = TokenBucket(capacity=10, rate=2)
for now, count in [(0, 12), (1, 3), (2.5, 1), (6, 4)]:
    passed = sum(bucket.take(now) for _ in range(count))
    print(f"t={now}: {passed} of {count} pass, {bucket.level:g} left")
# t=0: 10 of 12 pass, 0 left
# t=1: 2 of 3 pass, 0 left
# t=2.5: 1 of 1 pass, 2 left
# t=6: 4 of 4 pass, 5 left

This is part 2's bucket, letting the same 17 requests through. To limit tokens, pass each request's token count as cost and give the rate per second: 400,000 a minute is about 6,667 a second.

Check yourself

Four questions from across the lesson. Try each one before you look back.

1

A team's plan includes 2,000,000 tokens a month. By day 10 it has used 1,200,000. At the same pace, when does it run out?

1,200,000 ÷ 10 = 120,000 a day, and 2,000,000 ÷ 120,000 ≈ 16.7, so partway through day 17. It's only at 60%, so a warning set at 80% hasn't fired yet. Alerting on the projected run-out date catches fast burners earlier than a percentage does.

2

On Anthropic's API, each of your requests reads 10,000 input tokens, and 8,000 of them come from the prompt cache. How many count toward the input-tokens-per-minute limit?

Anthropic doesn't count tokens read from the cache toward that limit, so only the 2,000 uncached ones count. Caching the repeated part of the prompt lets five times as many requests through.

3

A limiter checks a request against the user, team and global limits in turn, and adds the request's tokens to each count as soon as that gate passes. What goes wrong?

A request that passes the user gate and fails at the team gate never runs, but its tokens were already added to the user's count. Check every gate first, then charge all three.

4

A team is at its per-minute cap. A live chat message and a step of a nightly batch job arrive together. What should happen?

A person is waiting on the chat message; nobody is waiting on the batch step this second. Giving interactive traffic priority, and letting background work fill the gaps, keeps the cap and the customer happy at the same time.

What to remember

  1. 1A budget at every level

    Budgets cap how much: a 200,000-token budget per task turns an $8,000 night of stuck loops into $400.

  2. 2Rate limits and the token bucket

    A token bucket's capacity sets the biggest burst and its refill rate the long-run average: at capacity 10 and 2 a second, 17 of the 20 bursty requests pass.

  3. 3Count tokens, not just requests

    Whichever limit runs out first throttles you: at 2,000,000 input tokens a minute, 10,000-token requests stop at 200 a minute, whatever the request limit says.

  4. 4Limits per user, per team and overall

    Admit a request only if the user, team and global gates all have room, then charge all three.

Recap and going further

The lesson on one page

Budgets
  • Per call, per task, per user, per month
  • Warn at 80%, then downgrade, then stop
Token bucket
  • Capacity sets the burst
  • Refill rate sets the long-run average
Count tokens
  • Requests, input and output tokens per minute
  • Whichever runs out first throttles you
Spending limits at every level
Three gates
  • User, team and global limits
  • Check all three, then charge all three
Refusing well
  • 429 with Retry-After
  • Live chat first, batch work queued

Try this

Find your account's real limits

Write a script that sends requests to a model API with a growing prompt (1,000, then 5,000, then 20,000 tokens) as fast as your account allows, and log the rate-limit headers from every reply. Then put a token bucket of your own in front of it, sized from those headers, and run the same script again.

You need: Python and an API key on a low usage tier (lower limits make the effect easy to see), plus a few dollars of credit.

Look for

  • Whether requests, input tokens or output tokens per minute run out first at each prompt size.
  • What the reset headers say, and how closely the refusals follow them.
  • How many 429s your own bucket prevents.
  • What the script's throughput looks like with and without your bucket.

The words, in plain language

Token budget
A ceiling on the tokens, or the money, something may spend: one call, one task, one user per day, the whole app per month. A request that would go over is stopped, moved to a cheaper model or held until someone confirms.
Token bucket
A rate limiter with two settings: a capacity, which sets the biggest burst, and a refill rate, which sets the long-run average. Anthropic's API uses one, so its limits refill continuously instead of resetting each minute.
RPM and TPM
Requests per minute and tokens per minute. Your account has both, often with input and output tokens counted apart. Whichever you run out of first is the one that throttles you.
Noisy neighbour
A customer whose heavy use slows down or blocks everyone else on shared capacity. A cap per team contains it: the bulk job just runs slower.

Watch

Read next

Next lessonExact and semantic cachesNext, the cheapest call of all: the one you never send, because the answer is already in a cache. Start lesson 4

Production AI overviewSpotted a mistake? Tell us.