What is CacheGap?
A BYOK (bring-your-own-key) LLM cost-optimization proxy. You point your OpenAI or Anthropic traffic at it, using your own provider API key. It restructures traffic to maximize the provider’s native prompt cache (prompt-cache orchestration) and proves the resulting savings per request, with an honest baseline.
Do you resell OpenAI or Anthropic access?
No, never. Your traffic always runs on your own provider key, and you pay the provider directly. CacheGap never holds a shared provider key and never marks up tokens. This is a hard constraint, not a pricing choice: reselling access violates provider terms of service.
Can CacheGap make my responses worse?
No. The default pipeline is lossless: provider prompt caching caches computation, not answers, so the model runs fresh inference every time and output quality is identical to not caching. Exact-match response caching returns a stored response only on a byte-identical request. Semantic (similarity-based) caching is a separate, opt-in, gated feature and is off by default.
Can CacheGap increase my provider bill?
The do-no-harm gate exists precisely to prevent this. Anthropic charges cache-write premiums (1.25x or 2x the input rate), so caching low-reuse traffic costs money. CacheGap emits cache writes only after observing that a route’s reuse clears the break-even threshold with statistical confidence. Routes that never clear it stay untouched, forever.
What is shadow mode?
Every new account starts in shadow mode: your requests are forwarded raw and untouched while CacheGap observes structure and reuse, then projects what optimization would save. You see the projected number, with a confidence figure, before a single request is modified. Activation is your explicit choice.
How do I know the savings numbers are real?
Every non-streaming response carries x-cachegap-baseline-usd, x-cachegap-actual-usd, and x-cachegap-savings-usd headers you can check against your provider’s own usage block and published pricing. Savings are reported net of cache-write premiums, scoped to input tokens, and attributed across what the provider’s automatic caching would have done anyway vs. what CacheGap added. The full methodology is public.
What happens if CacheGap has a bug or goes down mid-request?
The proxy fails open: any internal error degrades to forwarding your raw, untouched request to the provider. Optimization is best-effort; delivery is not.
What do you store? Is my provider key safe?
Provider keys are encrypted at rest with AES-256-GCM and used only for your own requests. The free audit tool runs entirely in your browser: the request you paste never leaves the page. Proxy-side, measurement persists token counts and cost metadata per request, not your prompt contents.
How much does it cost?
The free tier includes the audit tool, shadow mode, and the dashboard. Paid tiers are flat monthly subscriptions with token-based allowances, priced to be roughly 10 to 20 percent of dashboard-verified savings, with a guarantee: if verified savings in a month are below 2x your subscription, that month is free.
Which providers are supported?
OpenAI-compatible and Anthropic-compatible endpoints at launch: change the base URL, keep your SDK and your code. Streaming is fully supported.
Something missing? The methodology page covers how every number is computed, and the connect docs cover integration details.