What Is a Token Budget in LLMs?
A token budget is a limit on how many tokens a language model call, or a whole multi-step agent task, may use. The term covers three related limits: how a fixed context window is divided among the parts of a prompt and the reserved output, a cap on generated tokens including hidden reasoning, and a ceiling on total tokens across an agent's run for cost or latency control. A token budget governs how much goes into and comes out of a model, not whether that content is accurate.
Updated
What is a token budget?
Every model has a fixed context window, and the prompt and the response share it. Anthropic's docs define the window as "all the text a language model can reference when generating a response, including the response itself." Every prompt is therefore an allocation problem.
Context allocation. How much of the window each component gets, from system prompt to reserved output. The hard maximum is not the only limit. Anthropic's context engineering guide describes an "attention budget" and says "Every new token introduced depletes this budget by some amount." Its context window docs say some Claude models have "context awareness" and track "their remaining context window (their 'token budget')" during a conversation.
Output and reasoning. Reasoning models generate tokens the caller never sees. OpenAI's reasoning guide says these tokens "still occupy space in the model's context window and are billed as output tokens," that max_output_tokens caps everything generated, reasoning included, and that OpenAI "recommends reserving at least 25,000 tokens for reasoning and outputs" when starting out. Anthropic's manual extended-thinking mode, which newer Claude models replace with adaptive thinking, takes a budget_tokens value that its docs call "a target rather than a strict cap." Han et al. found that chain-of-thought reasoning "can be compressed by including a reasonable token budget in the prompt," though the choice of budget "plays a crucial role" in how well that works.
Task budgets. An agent makes many calls, so teams also budget a whole run. Anthropic's task budgets (in beta) are advisory: they "let you tell Claude how many tokens it has for a full agentic loop, including thinking, tool calls, tool results, and output."
How a token budget works
An illustrative allocation for one agent step on a model with a 200,000-token window:
| Part of the prompt | Tokens | When it runs over |
|---|---|---|
| System prompt and policies | 5,000 | Shorten offline; fixed at run time |
| Tool definitions | 10,000 | Load only the tools this step needs |
| Conversation and tool history | 70,000 | Offload large tool results, then compact older turns |
| Retrieved documents | 45,000 | Rerank and keep fewer passages |
| Current task and inputs | 10,000 | Rarely trimmed |
| Reserved for reasoning and output | 60,000 | Never borrowed by the input |
The harness counts tokens before each call and trims in order of least harm: unneeded tools, then bulky tool results, then old history, then retrieval. The output reserve comes first, because running out of it fails badly. When generation hits the limit, OpenAI's API returns a response with status incomplete, and this "might occur before any visible output tokens are produced," so the caller can pay for input and reasoning tokens and get no visible answer (OpenAI).
Why token budgets matter in real-time systems
Budget pressure pushes teams to prepare context ahead of time: a nightly account digest, a cached customer profile, a compacted history. Each saves tokens, and each is a derived copy with an age. That is safe for a static document. For mutable state it trades prompt size for staleness: the digest fits the budget and describes yesterday.
A token budget and a freshness requirement are separate constraints; meeting one says nothing about the other. The facts a decision gates on, such as a balance, a limit or an order status, are usually small. Read those fresh at call time and spend the budget on bulky, slow-changing material. The wider discipline is laid out in what context engineering actually means.
FAQ
Related terms
Context compression reduces tokens in a language model's context while keeping what the task needs, by summarizing, pruning, trimming or compressing prompts.
Context offloading moves information out of an LLM's context window into external storage, such as files or notes, and pulls it back only when a step needs it.
Context isolation splits an agent's work across separate context windows, such as sub-agents, so each model call sees only what its part of the task needs.
Lost in the middle is the tendency of language models to use information at the start or end of a long prompt better than information placed in the middle.
A freshness budget is the maximum staleness a decision can tolerate in its inputs — and it equals the worst input, since one slow signal spends it all.