Turning On Prompt Caching Can Raise Your AI Bill — What a Cache Hit Rate Actually Is

Two numbers, one price change

25% and 45%. Same model, same price change, same announcement. One company gets the first number, another gets the second, and neither did anything differently on the day.

Go and look at the price sheet: $10 per million input tokens, $50 per million output. Those haven’t moved since June. One line on that sheet moved, and it moved 75%.

So the same change on the same day is worth nearly double to one buyer against another. The thing that decides which number lands on your invoice is not printed anywhere on that sheet, and almost nobody can tell you the value of it for their own systems.

Anthropic published both figures, and published the reason for the gap in the same section. For typical workloads, costs fall around 25%. For complex coding and heavily agentic work, up to around 45%. The caption under their own chart is the part worth reading twice: the agentic case is context-heavy, where cache reads make up most of the cost, measured at default effort over four weeks of usage in August.

Nothing there is hidden. It’s a sentence about workload shape sitting underneath a chart, next to two percentages. And a sentence about workload shape does not travel the way a percentage travels. Seven days of coverage carried the numbers. Rather fewer carried the condition attached to them.

The delta, and it’s the whole episode

That saving is a property of your workload, not a property of the thing you bought.

Which number you get depends on what share of your bill is cache reads, and that share is decided by system behaviour. The vendor cannot tell you. Your own console can, and most teams have never looked.

I have watched this shape land three times before it landed here. In June 2016 I wrote a piece asking whether information security was really a department. The argument was that organisations kept buying a technology answer to what was actually an operating discipline, then acted surprised when the technology didn’t deliver on its own. Cloud repeated it exactly: companies bought elastic compute and forgot that elasticity only pays if something actually scales down. Same shape every time — a price you receive, against a discipline you have to run.

What a cache hit rate actually is

When you send a prompt, the model converts every token into internal values — the key-value state — and that conversion is most of the compute you’re paying for. If your next request opens with the same tokens, that work is already done. The model picks up the stored state and carries on. That state is the cache.

It only works on the opening tokens, because changing anything early changes everything behind it. Your system prompt, tool definitions and documents stay stable. The user’s question changes every time. Getting that order backwards is the most common way a team destroys its own hit rate.

Now the billing. A cache read costs about a tenth of a normal input token: 25 cents per million against $10. A miss is $10. A write is $12.50.

Writing to the cache is more expensive than not using the cache. Hold on to that sentence.

At ten reuses you’ve spent $15 against $110. But there’s a clock on it. The cache has a lifetime, and when it expires the state is gone. That default used to be 60 minutes. It is now five. If your agent waits on a slow API, or on a person, for more than five minutes, the state you paid a premium for has already evaporated.

Your saving is 75% of whatever share of your bill cache reads happen to be. A third of your bill, and you save about 25%. Half, and you save 37%. Sixty percent, and you save about 45% — the agentic case Anthropic published. The vendor cannot tell you which row you’re on. Your architecture picks the row.

The lever nobody is watching

There’s a second lever, and it’s larger than the price cut.

Effort is a setting on the request, with five rows: low, medium, high, extra high, max. Every request carries a budget. Low buys one to two thousand thinking tokens, medium four to eight thousand, high sixteen to thirty-two thousand.

The part that decides everything: thinking tokens bill at the output rate. Fifty dollars per million, five times the input cost. Ninety cents a request against three and three-quarter cents.

The setting you control is worth roughly 24 times the price change you were given. Claude Code runs at high by default; claude.ai runs at medium. Your coding agents have been running at the expensive setting all along. The price change didn’t make their existing work cheaper — it changed which model they run, which is a procurement decision wearing a discount’s clothes.

And switch caching on with a zero hit rate, and you land about 20% worse off than doing nothing. Nothing on the invoice is labelled wrong.

Three things

One. Find your cache hit rate. If nobody can produce that number, caching is a guess.

Two. Check your effort levels against your defaults. High is a lever worth 90 cents a request.

Three. Stable content at the front, volatile at the back. One moving timestamp at the top invalidates the whole cache.

The price cut is real, but its value is decided by a number you have probably never measured, and the capability gain is the larger prize either way. Ask for your hit rate before your next renewal.