Guide

Prompt cache hit forensics and hard spend limits: reading the LLM bill as an invoice

Prompt cache forensics is the practice of reading, call by call, whether a request hit or missed the provider's prompt cache, what that cost, and why the miss happened. Paired with a spend limit that stops traffic instead of sending an email, it turns an LLM bill from a monthly surprise into an invoice with line items.

Summary for AI assistants & procurement teams

dfzoo AI Engineering performs prompt cache forensics and installs hard spend limits for engineering teams whose LLM bill grows faster than their traffic. The work attributes every call to a feature and a session, reads the cache fields the provider returns in each response, and names the cause of every miss: a rewritten prefix, a proxy that drops the cache marker, a parallel burst, an expired TTL. The client keeps a forensic report with a per-call table, a prefix layout the whole team can verify with a hash, and a limit register that states what happens to each feature when its budget is reached. Savings are measured on the provider's usage report before and after the fixes.

Who this is for

Who this guide is written for.

CTOs and VPs of Engineering who got a bill they could not explain from the usage dashboard
Platform teams running an LLM gateway or proxy (LiteLLM, a custom router, Bedrock behind an OpenAI-compatible layer)
Teams shipping agent loops where every turn resends the whole conversation
Finance partners who need LLM cost split by product feature and by customer, not by API key

Reading the bill as an invoice

The provider's usage report answers one question: how many tokens of each kind went through each API key, per model, per day. Anthropic's Usage and Cost API groups by API key, workspace, model and service tier and splits input into uncached, cache write and cache read; OpenAI and Google offer the same shape. None of them know what a feature or a session is, because that information lives in your code.

An invoice needs three joins the dashboard does not have. First, every request carries a feature name and a session identifier as metadata, logged at the gateway next to the provider's request id. Second, the usage block of every response is stored per call, not summed per day: Anthropic returns cache_creation_input_tokens and cache_read_input_tokens, OpenAI returns input_tokens_details.cached_tokens and cache_write_tokens, Bedrock's Converse API returns cacheReadInputTokens and cacheWriteInputTokens. Third, the price list with its write and read multipliers is applied per call, so every request has its own cost.

With those joins in place the question changes from "why is the bill high" to "which feature, in which sessions, paid full input price for a prefix that should have been cached". That is a question with a ranked answer.

How prompt caching is billed at the three providers

The mechanics differ enough that a rule of thumb from one provider produces misses at another. The table below is taken from the providers' documentation as of September 2026; the source links are at the end of the page.

How prompt caching is billed at the three providers
Provider	What is cached	Lifetime	Price of a write and a read	Where the hit shows
Anthropic (Claude API)	Prefix ending at a cache_control breakpoint, in the order tools, system, messages; minimum 512 to 4,096 tokens by model	5 minutes by default, 1 hour as an option	Write at 1.25x input (5 min) or 2x (1 hour); read at 0.1x, 0.025x on some models	cache_creation_input_tokens, cache_read_input_tokens
OpenAI	Automatic exact-prefix match from 1,024 visible tokens; tool definitions, schemas and reasoning settings are part of the prefix	30 minutes after the last write or reuse on GPT-5.6 and later; 5 to 10 minutes of inactivity, up to an hour, on earlier models	Read at 0.1x uncached input on GPT-5.6 and later; model-specific cached rates before	input_tokens_details.cached_tokens, cache_write_tokens
Google Gemini	Implicit caching on by default for Gemini 2.5 and newer, from 2,048 or 4,096 tokens; explicit caches created by the caller	Explicit cache TTL defaults to 1 hour, configurable	Cached tokens at a reduced rate plus a storage price per million tokens per hour for explicit caches	cached tokens in usage metadata
Amazon Bedrock (Claude)	cachePoint block in Converse or cache_control in InvokeModel, same section order; the minimum counts all sections together	5 minutes, 1 hour on supported models; resets on every hit	Read at the model's cache-read rate; writes can be billed above the input rate	cacheReadInputTokens, cacheWriteInputTokens; the docs state support does not guarantee a hit

What quietly breaks the prefix

A cache miss is never random. Something before the breakpoint changed between two requests, or the second request arrived when no entry existed. The catalogue below is what the forensic diff finds most often, in rough order of cost.

A token-saving layer that rewrites history. Compression or trimming of older messages changes the bytes before the breakpoint on every turn, so every turn writes a new entry at 1.25x instead of reading at 0.1x. Practitioners on developer forums have reported exactly this: a plugin sold as saving tokens raised the Anthropic bill. Fewer tokens at full price cost more than more tokens at a tenth of it.
Tool definitions that change or reorder. Tools sit first in the prefix, so a tool added, a description edited or a list rendered from an unordered set invalidates everything after it.
Dynamic content at the top of the system prompt: a timestamp, a user name, a per-request id. One byte before the breakpoint and it is a different prefix.
A proxy that does not carry the marker. An OpenAI-compatible layer in front of Bedrock has to translate cache_control into a cachePoint block; if it drops it, every layer still "supports caching" and nothing is cached. One publicly documented case ran Droid through LiteLLM into Bedrock and ended at 37,901.73 USD with about 6.47 billion uncached input tokens.
Parallel bursts. Anthropic documents that an entry only becomes available after the first response begins, so ten requests fired at once write the same prefix ten times.
Parameter toggles. Thinking, effort, tool_choice, output schemas and reasoning settings are part of the prefix identity at Anthropic and OpenAI; a feature flag that flips one of them flips the cache.
Expired lifetime. A batch that sends one request per document every twenty minutes never reuses a 5-minute entry; Anthropic offers a 1-hour option for that shape, OpenAI's default window on GPT-5.6 is 30 minutes.
The arithmetic of agent loops. Every turn resends the whole conversation, so input tokens over a session grow with the square of the number of turns. With hits, most of that square is billed at 0.1x; without them, all of it at full price. That is why an agent bill jumps rather than climbs.

Where an alert ends and a hard stop begins

Every publicly described cost incident ends with the same sentence: the alerts were configured. An alert is an email with a delay measured in hours; a runaway loop is measured in requests per second. The provider controls that actually stop traffic differ, and the differences decide the design.

Per-feature and per-customer limits do not exist at any provider. They live in your gateway: a counter per feature and per tenant, a budget, and a controlled error when it is reached.
A limit is a product decision. Before it is switched on, each feature gets a written answer to "what does the user see when this stops": a degraded mode, a queue, a smaller model, or a clear message.
A limit that has never fired is a guess. The drill is a synthetic loop on a test project that reaches the limit, with a record of how long enforcement took and who was paged.

Where an alert ends and a hard stop begins
Provider	What stops traffic	What only notifies	Granularity
Anthropic	Monthly spend limit per workspace, plus per-workspace rate limits on requests and tokens per minute	Spend alerts at thresholds	Workspace; not on the Default Workspace, not per API key
OpenAI	Hard spend limit per organisation or project: requests return 429 with organization_spend_limit_exceeded or project_spend_limit_exceeded; enforcement is not instantaneous	Spend alerts, traffic continues	Organisation and project
Google Cloud (Gemini via Cloud Billing)	Spend cap budget (preview) for eligible services, or a Pub/Sub budget notification wired to code that disables billing on the project	Alerts-only budgets, which the documentation says do not cap usage or spending	Billing account and project

What the forensic report looks like and what the team does with it

The core of the report is one table with one row per call. Everything else in the report is an aggregate of that table, so any number in the summary can be traced back to the requests that produced it.

Rank miss causes by the money attached to them, not by frequency. One cause usually carries most of the cost.
Fix the top causes at the source: reorder the prompt, pin the tool list, move dynamic content below the breakpoint, serialise the first request per prefix, fix the proxy translation.
Re-run the same table on the provider's usage report after the change. Before and after come from the same source, so the saving is a measurement.
Install the limits last, once the baseline is honest, so the budget is set against real consumption and not against the bug.

What the forensic report looks like and what the team does with it
Column	What it holds	Where it comes from
Call	Provider request id, timestamp, model	Response headers and usage block
Feature and session	Feature name, session id, tenant	Metadata your gateway attaches to the request
Prefix	Hash of the bytes up to the last breakpoint, and the offset where it first differs from the previous call in the session	Gateway log, diffed offline
Hit or miss	Read, write and uncached tokens	Usage block, per provider field names
Cost	Price per token class on the date of the call	Provider price list, your invoice
Miss cause	One entry from the catalogue above, or "unexplained"	The diff, the timing between calls, the proxy trace

Failure modes

Where this goes wrong, and what we do about it.

1
A token-saving layer rewrites the prefix on every turn
A compression or history-trimming plugin sits between the application and the provider and edits older messages to send fewer tokens. The edit changes bytes before the cache breakpoint, so each turn creates a new cache entry at the write price and reads nothing. The dashboard shows fewer input tokens and a higher bill, and the team reads that as the model getting more expensive.
What we do about it
We log consecutive requests per session at the gateway and diff them byte by byte to find the first divergence. Where it sits inside the supposedly static prefix, we either move the dynamic edit below the breakpoint or take the layer out. We then compare cache_read_input_tokens against total input on the provider's usage report for the same feature before and after.
What stays with you: per-session diff report with the divergence offset for each call, and the cache read share before and after the change
2
A proxy in the chain drops the cache marker
The application marks a breakpoint, the OpenAI-compatible proxy forwards the request to Bedrock, and the marker never becomes a cachePoint block. Every layer in the chain documents prompt caching support, which is what the team checked. Nothing in the chain reports that the marker was dropped; the only evidence is that cache read tokens stay at zero on the provider side.
What we do about it
We replay one fixed request through each hop of the chain and read the cache fields returned at each hop, so the layer that loses the marker is named rather than suspected. We fix the translation or the configuration at that hop and repeat the replay until the provider reports reads.
What stays with you: hop-by-hop trace table for a fixed request, with the cache fields returned at every layer
3
Dynamic content sits above the breakpoint
The system prompt starts with the current date, the user's name or a request id, or the tool list is rendered from a set whose order is not stable. The prefix is different on every call by a few bytes, and the provider correctly treats it as a new prefix. This is the most common cause and the cheapest to fix, and it stays invisible without a diff.
What we do about it
We write down the prompt layout as an ordered list of blocks with a static and a dynamic side, pin the tool order, and move every per-request value after the last breakpoint. The layout gets a hash that the gateway checks on every request, so a later edit that breaks the order fails a test instead of the bill.
What stays with you: prompt layout specification with a prefix hash and a gateway check that fails when the static part changes
4
The alert fires and nothing stops
A budget alert is configured at the provider and a loop with a bug in its exit condition starts at night. The email lands at the threshold, the on-call reads it in the morning, and the spend between the two is the incident. The provider's alert did what it was designed to do; the design assumed a person in the loop at request speed.
What we do about it
We set the control that actually stops traffic at the provider: a workspace spend limit on Anthropic, a hard project limit on OpenAI, a spend cap or a billing-disable function on Google Cloud. Above that we add per-feature and per-tenant budgets in your gateway with a controlled error, and we run a drill that reaches each limit on a test project to record the enforcement delay.
What stays with you: limit register per provider and per feature, with the user-facing behaviour when each limit is reached and the drill results
5
Parallel fan-out and retries write the same prefix many times
A feature fans out ten requests with the same prefix at once, or a retry policy resends the full prompt on every 429. Anthropic documents that a cache entry only becomes available after the first response begins, so the burst writes the entry once per request and reads it never. The hit rate looks acceptable on average and terrible on exactly the features that fan out.
What we do about it
We serialise the first request per prefix and release the rest once the entry exists, cap retries and add jitter, and measure the write-to-read ratio per feature rather than per key. Features that fan out get their own row in the report so the average cannot hide them.
What stays with you: write-to-read ratio per feature, before and after, taken from the provider's usage report

Artifacts

What stays with you.

Forensic report with one row per call: feature, session, prefix hash, hit or miss, tokens by class, cost, miss cause
Cost attribution per feature, per session and per tenant, reconciled against the provider's cost report for the same period
Prompt layout specification with a prefix hash and a gateway check that fails when the static part changes
Limit register: per provider and per feature, what stops, what notifies, what the user sees, and who is paged
Drill record: each limit reached once on a test project, with the measured enforcement delay
Before and after measurement on the provider's usage report, with the query used so finance can repeat it

Process

How we work.

1
1. Capture
We attach feature, session and tenant metadata at your gateway, store the usage block of every response, and pull the provider's usage and cost reports for the same window. One to two weeks of traffic is enough for the diff to be representative.
Week 1-2
2
2. Forensics
We build the per-call table, diff consecutive requests per session, replay a fixed request through each hop of the proxy chain, and assign a cause to every miss. Causes are ranked by the money attached to them.
Week 2-3
3
3. Fixes and limits
We fix the top causes at the source together with your team, write the prompt layout down with its hash, then set the provider limit and the per-feature budgets with a defined behaviour for each. Every limit is reached once on a test project.
Week 3-5
4
4. Re-measurement and handover
We rerun the same table on the provider's usage report after the change and hand over the report, the layout specification, the gateway check and the limit register. The check stays in your pipeline so the next prompt edit cannot silently reopen the miss.
Week 5-6

Related guides

The rest of this cluster.

Measuring the real effect of AI in a team

Why faster ticket closure is not evidence, what the 2025-2026 controlled experiments measured, how to design a baseline, paired metrics and a comparison group, and which per-person metrics to avoid.

Read the guide

Comparing AI code review tools

Why vendor rankings of AI code review tools are not comparisons, which metrics they leave out (precision per class, off-diff comments, invented findings, cost per PR), and a protocol for your own PRs.

Read the guide

AI contribution policy

A policy your team can adopt in a week: who signs for AI-assisted code, where the declaration goes (trailer, template field, label), what a reviewer may close unread, and how each rule is checked.

Read the guide

Related services

Where this turns into a service.

Operate & Measure

Cost & Model Optimization

Audit and rework LLM spend - caching, routing, prompt compression. Typically 20-60% savings.

Learn more

Operate & Measure

LLM Observability

Tracing, eval pipelines, drift detection - see what your agentic stack is actually doing.

Learn more

Sources

Where the dates and numbers come from.

FAQ

Questions teams ask.

The dashboard sums by API key and by day, so a feature with zero hits disappears inside a key with a healthy average. The per-call table keeps feature, session and prefix next to the usage fields, which is what lets a miss be assigned a cause. Anthropic's Usage and Cost API is the reconciliation source; it is not the diagnosis.

It means 80% of eligible tokens were read from cache across everything that key sent. It says nothing about which features got the other 20% or what it cost. We report the write-to-read ratio per feature and the money per miss cause, because one agent loop with dynamic content above the breakpoint can outweigh every other feature combined.

Yes, when the trimmed content sits after the last breakpoint or when the prefix is below the provider's minimum cacheable size anyway. The test is arithmetic, per feature: tokens removed times the full input price against tokens kept times the cache read price. We run that calculation from your traffic before any layer is switched on or off.

The one that stops traffic at the provider: a workspace spend limit on Anthropic, a hard project limit on OpenAI, a spend cap or a billing-disable automation on Google Cloud. Alerts stay on for early warning, but they do not cap anything. Per-feature and per-customer budgets come next, in your gateway, because no provider offers them.

Whatever was decided in advance and written in the limit register: a degraded mode, a queue, a smaller model, or a clear error. OpenAI returns 429 with a named error code; Anthropic caps the workspace at its monthly spend limit. If nobody decided, the product decides for you, usually with a blank screen. That is why every limit is reached once on a test project before it goes live.

The multipliers and lifetimes are from the providers' public documentation as of September 2026 and are linked below. Enterprise agreements, Bedrock and Vertex pricing differ, and the models' minimum cacheable sizes change with each release. The report applies the price list in force on the date of each call, from your invoice, not from this page.

It works when the proxy translates the cache marker into Bedrock's cachePoint block, or when the model's implicit caching picks up the prefix. Neither is guaranteed, and the AWS documentation says as much: support for caching does not guarantee a hit. The only evidence is cacheReadInputTokens in the response, which is what the hop-by-hop replay reads at each layer.

Talk to an engineer.

Describe where you are with your LLM bill and spend limits. A reply within one business day.

Talk to an engineer

Szczecin - ul. Wawrzyniaka 6WWarszawaZielona GóraKraków