blog / realized-discount-rate
Realized Discount Rate: why your 90% prompt-caching discount is not what you actually get
Every major provider now advertises a headline discount on cached input tokens: up to 90% on Anthropic cache reads, up to 90% on OpenAI's GPT-5.x family, roughly 90% implicit on Gemini 2.5+. Read those numbers and it sounds like your input bill should be near zero. Then you look at your invoice and it is not. The number that explains the difference is what we call the Realized Discount Rate.
Definition
RDR is your effective input cost divided by list input cost, over a real window of traffic:
RDR = 1 − (actual input spend / (total input tokens × list input rate))
If you processed 100M input tokens on a model listing at $3.00/MTok, list cost is $300. If your actual input spend was $210, your RDR is 30%. The provider's headline says 90% is available on cached reads; you realized 30%. That 60-point gap is not a rounding error, it is the single largest addressable line on most LLM bills.
Why the gap exists
The headline discount applies only to tokens that are actually served from cache, and cache hits are prefix-based: anything that changes early in the prompt invalidates everything after it. In practice the gap comes from a short list of structural causes:
- Dynamic content sitting mid-prompt. A timestamp, a user name, or working memory injected before the stable instructions breaks the prefix on every request. ProjectDiscovery's public teardown of their Neo platform is the canonical example: moving dynamic content to the tail took their hit rate from 7% to 74% in one deploy.
- Unstable tool schemas. Agent frameworks that reorder or re-serialize tool definitions per request produce byte-different prefixes that are semantically identical and cache-hostile.
- TTL mismatches. Traffic that recurs every 20 minutes against a 5-minute cache pays write premiums repeatedly and reads never.
- Sub-minimum prefixes. Anthropic's minimum cacheable block is about 1,024 tokens; short prompts are simply not in the game, and any tool that claims savings on them is lying to you.
The number nobody shows you
Provider dashboards show usage. They do not show what your workload could realize with the same models and the same traffic under better structure, which means they cannot show you the gap. That takes an achievable rate: an estimate, computed from your actual request shapes, of the discount a correctly structured version of the same workload would realize. RDR vs. achievable is the honest framing of prompt caching: not "turn on caching, save 90%" but "you are realizing X, your traffic could realize Y, and the difference is $Z/month."
Two honesty rules when you compute this yourself
First, scope everything to input tokens. Caching does not touch output or reasoning tokens, which are a growing share of agentic spend. An "80% savings" claim that ignores the output share of your bill is theater; always compute and show the input/output split. Second, subtract write premiums. Anthropic bills cache writes at 1.25x (5-minute TTL) or 2x (1-hour TTL) of the input rate, so low-reuse traffic can realize a negative discount: caching that raises your bill. Net, or it does not count.
How to measure your RDR today
You need three numbers per window: total input tokens, cached-read input tokens, and actual input spend. All three are derivable from the usage blocks your provider already returns. Divide as above, then compare against what a prefix-stable restructuring of your top routes would plausibly hit. If computing that counterfactual by hand sounds like work, that is the part we automated.