Prompt Caching & Cost

When every request to a model starts with the same tokens — the same system prompt, the same tool definitions — the provider can run the model over that prefix once, store what the run produced, and start every later request where the prefix ends. What is stored is the prefix’s keys and values: the per-token vectors that attention reads. Because a token’s keys and values depend only on the tokens before it, the stored entries are the same numbers a fresh computation would produce — reuse is exact, not an approximation. Prompt caching is that reuse, sold: a cached input token is billed at a fraction of a fresh one, and for an agentic loop, whose every turn resends a long and mostly unchanged transcript, the cache is the difference between linear and quadratic spend over a session.

The stored computation is the prefill: the pass over the prompt that runs before the first output token appears. A cache hit skips the prefill for the covered prefix, so it cuts time-to-first-token as well as cost. What the prefix contains in the first place — tool definitions, then the system prompt, then the transcript — is anatomy of a request’s subject.

An exact prefix, or nothing#

A cache entry is keyed on the exact bytes of the rendered prompt, and a request reuses it only for the longest stored prefix that matches byte for byte. One changed early token recomputes everything after it, for two mechanical reasons, each derived on its own page. First, every later token’s keys and values depend on the changed token, so none of them can be kept. Second, position is baked into the stored vectors: under rotary position embedding (RoPE), a key is rotated by its absolute position when it is first computed and stored already rotated, so even text that reappears verbatim at a different offset produces different keys — there is no salvaging a match that starts one token later.

The consequence for prompt layout: volatile content early is a standing cache miss. A current-date line at the top of a system prompt turns the first request of each day into a full re-prefill of everything below it; a per-request ID in the same place does it on every request. The practice that follows — stable content first, volatile content last — is a context-engineering habit whose entire justification is this paragraph. For a concrete instance — why keeping CLAUDE.md and tool schemas stable at the front of the window makes a session cheap — see the section’s running example, Claude Code: writing a page for this wiki.

A cache is also scoped to one model. The keys and values are what that model’s weights compute; the same text run through different weights produces different numbers, so switching models mid-session forfeits the cache even when the transcript is identical.

What the vendors sell#

Anthropic caches at explicit breakpoints: cache_control markers, at most four per request, placed on content blocks. The prompt renders in the order tools, then system, then messages, so a marker on the last system block covers the tool definitions with it. An entry lives five minutes by default or one hour as a paid option, and a read refreshes the timer, so requests arriving inside the window keep an entry warm indefinitely. Writes are billed at a premium — 1.25× base input for the five-minute lifetime, 2× for the hour — and reads at roughly a tenth of base, varying by model. A prefix below a model-dependent minimum — on the order of a thousand tokens — silently doesn’t cache at all (Anthropic’s documentation carries the current numbers).

OpenAI caches automatically: no markers, any prompt of 1,024 tokens or more is eligible, and cached tokens are billed at a reduced per-model rate — up to 90% off. The mechanics have shifted between model generations: earlier models matched in 128-token increments and evicted entries after five to ten idle minutes, an hour after last use at most, while current ones keep a prefix reusable for thirty minutes after its last use and charge a write premium as Anthropic does (OpenAI’s guide carries the current rules).

Self-hosted serving stacks do the same without a bill. vLLM’s automatic prefix caching reuses stored blocks across requests, and SGLang’s RadixAttention (Zheng et al. 2023) keeps every live prefix in a radix tree — a tree whose branches are shared string prefixes — so a new request reuses the longest prefix any earlier request left behind, not just a designated one.

The arithmetic of a hit#

The five-minute entry pays for itself at the second request: a write plus one read costs 1.25× + 0.1× = 1.35× of base, against 2× for prefilling the prefix twice. The one-hour entry costs more to write than it saves on one read — 2× + 0.1× against 2× — and pays from the third request on; it exists for traffic whose gaps outlive the five-minute timer, not for steady loops.

The quadratic claim in the lede is the same arithmetic run over a session. An agent’s turn n resends a transcript roughly n messages long, so an uncached session prefills 1 + 2 + … + n turns’ worth of tokens — growth with the square of the session length. With a cache each token’s prefill is paid about once, plus the read fee: linear. On a long session the ratio between those two curves, not the per-token discount, is what decides whether the loop is affordable.

What caching does not save#

Three costs survive a hit, all derived on the KV cache page:

A 20,000-token system prompt served from cache is billed at a tenth and still costs full attention arithmetic on every token of every reply. Caching changes what a prefix costs to load, never what it costs to carry.

A cache can be observed from outside#

A hit returns faster than a miss, and that timing difference leaks. Gu et al. (2025) audited seventeen commercial APIs by timing probe prompts: eight providers were caching, and at seven of them — OpenAI among them, at audit time — entries were shared globally, so a probe returning fast revealed that some other user had recently sent the same prefix. Sharing a cache across customers is sharing a side channel. The major vendors now document scoping instead: Anthropic states that caches are never shared across organizations, and further isolates them per workspace on its own platform; OpenAI states the same at the organization boundary — trading some hit rate for not answering timing questions about other people’s prompts.

Beyond caching: batches and routing#

Two other levers move the same bill. Batching trades latency for price: Anthropic’s Message Batches and OpenAI’s equivalent take a queue of requests with no interactive deadline at half price — OpenAI completes each batch within twenty-four hours, Anthropic finishes most within an hour and expires anything still unprocessed at the day mark without billing it. The discount buys the provider the right to schedule the work into idle capacity. Routing picks a cheaper model per request. It interacts badly with caching — caches are model-scoped, so a session that bounces between models re-prefills at every switch — and the honest metric is cost per completed task, not per request: a cheaper request that needs more turns to finish the job isn’t cheaper. Which builders publish which prices is on the model builder pages.

Sources#

Backward Law and Ethics Agentic Workflows Forward