blog / prompt-caching-for-agents
Prompt caching for agent loops: why agents miss the cache and how to fix it
Agents should be the best-case workload for prompt caching: a large stable scaffold (system prompt, tool definitions, skills, retrieved context) re-sent on every step of every loop, often tens of thousands of tokens deep. On a 100k-token stable prefix with 2k dynamic input, steady-state caching cuts per-call input cost by roughly 84% at Sonnet-class rates. In practice, agent workloads routinely miss the cache on most steps. The causes are specific and fixable.
Failure mode 1: tool schemas that will not sit still
Prefix caching matches bytes, not meaning. Frameworks that rebuild tool definitions per request (dict ordering, optional fields appearing and vanishing, float formatting) produce a semantically identical, byte-different prefix every time, and the cache misses on the largest stable block you have. Fix: serialize tool schemas once, deterministically, and treat any change to them as a deliberate deploy, not a per-request accident.
Failure mode 2: working memory injected mid-prompt
Scratchpads, plans, and retrieved context that get spliced between the system prompt and the conversation invalidate everything after them on every step. Fix: relocate volatile state to the tail of the prompt, after the stable scaffold and history. This is the relocation trick, and it is the single most valuable structural change on agentic traffic.
Failure mode 3: mutating history instead of appending
Multi-turn caching works when each turn extends the previous prefix. Loops that rewrite earlier messages (summarizing them in place, trimming from the middle, re-scoring tool results) break the chain. Fix: append-only history within a session where possible; if you must compact, compact at deliberate boundaries and expect to pay one re-write there.
Failure mode 4: TTLs shorter than your loop cadence
An agent that acts every few seconds keeps a 5-minute cache warm effortlessly. A workflow that resumes every 20 minutes does not, and on Anthropic it pays a write premium each cold start. Whether longer TTLs pay depends on measured recurrence (break-even math here); on OpenAI, retention up to 24 hours on newer models makes slow loops cacheable that never used to be.
Failure mode 5: nobody is measuring per route
An agent product is many routes in a trench coat: planner, executor, summarizer, each with its own prefix shape and recurrence pattern. Aggregate hit rate hides the one route that is burning the budget. Track cached vs. fresh input tokens per route from the usage blocks you already receive, and compute your realized discount against list price (method here).
The compounding effect
Agent loops multiply everything: one structural fix on the scaffold pays on every step of every task, and one mid-prompt timestamp costs on every step of every task. That is why agentic workloads show both the best realized discounts in the field and the worst, on the same models, at the same list prices. Structure decides which side you are on.