A budget for the context window

Prompting, lesson 7

Decide what earns a place in the prompt and what stays out.

You know a model only sees what's in its prompt, and that a longer prompt costs more memory, time and money ("The price of a long context" in Inference and Serving).

Part 1. Everything competes for space

A help bot for a photo-editing app is about to answer a question. Here's everything it could put in the prompt.

Example 1.1

Everything the bot could send

system prompt500
the new message50
The parts that must go in: 550 tokens.
12 tool descriptions×150=1,800
Every tool the bot can use, described so it knows how to call each one.
40 earlier turns×120=4,800
The conversation so far.
30 help articles×400=12,000
Every help article that might apply.
550+1,800+4,800+12,000=19,150
19,150 tokens. Say your budget is 4,000 tokens per call, to keep replies quick and cheap. That's nearly five times over.

Your turn

The model's window holds 128,000 tokens, so all 19,150 fit. Why not send everything?

You pay for every input token on every call, and a longer prompt takes longer to process. Worse, extra text isn't neutral. A 2023 study added one irrelevant sentence to grade-school maths problems, and model accuracy dropped clearly. A help article about a similar feature can do the same to a support answer.

Show the answer

A. You pay for every input token on every call, and a longer prompt takes longer to process. Worse, extra text isn't neutral. A 2023 study added one irrelevant sentence to grade-school maths problems, and model accuracy dropped clearly. A help article about a similar feature can do the same to a support answer.

Context budget

The number of tokens you allow yourself per call, usually far below the window. Deciding what fills it is often called context engineering.

In practice

Count tokens with the model's own tokenizer, not by characters or words: tiktoken for OpenAI models, or the provider's token-counting endpoint. The same text can come to quite different counts in different models.

Part 2. Relevance over recency

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Relevance over recency
Some things always go in
Near misses do the most harm

See Pro plans I have Pro: sign in

Next lessonOrder effects in long promptsWhere you put information changes whether the model uses it. Start lesson 8

Prompting overviewSpotted a mistake? Tell us.

07 · A budget for the context window