Prompt caching math: when it saves 75% and when it costs you more

Prompt caching sounds like free money. Mark the stable start of your prompt as cacheable, and later requests that begin with the same tokens are billed at a steep discount. In practice I have seen teams turn it on and watch their bill go up. Both outcomes follow from the same three numbers.

The pricing shape

Providers differ, but a common structure looks like this, relative to the normal input price:

• Cache write: about 1.25x. You pay a premium the first time a prefix is stored.
• Cache read: about 0.1x. Later requests that reuse the prefix pay a tenth.
• Time to live: a few minutes, refreshed on each hit. Idle prefixes expire.

We will use $3 per million input tokens as the base. Output tokens are unaffected by caching, so we leave them out.

A worked example

An agent with a long system prompt and tool definitions: a 20,000-token stable prefix, plus about 1,000 tokens of conversation that changes every turn.

const BASE = 3 / 1e6;
const prefix = 20_000, variable = 1_000;

const uncached = (prefix + variable) * BASE;                 // $0.063
const miss = prefix * BASE * 1.25 + variable * BASE;         // $0.078
const hit  = prefix * BASE * 0.10 + variable * BASE;         // $0.009

const blended = (h: number) => h * hit + (1 - h) * miss;

A hit costs about a seventh of an uncached call. A miss costs about 24% more. So the question is entirely about your hit rate.

Solving for break-even: 0.078 minus 0.069 times the hit rate equals 0.063, which gives about 22%. Below that, caching loses money. Above it, savings climb fast. At 100,000 calls a day, the difference between 0% and 90% is roughly $7,800 a day against $1,590.

Why hit rates stay at zero

Caches match on an exact prefix. One different token near the top and everything after it is a miss. The usual culprits:

• A timestamp in the system prompt. “Current time: 14:32:07” changes every second. Move it to the end of the prompt, or round it to the day.
• Per-user details early on. Put the user’s name and plan after the shared instructions, not before them.
• Unstable tool order. If tool definitions come from a map or set, their order may change between requests. Sort them.
• Non-deterministic JSON. Serializing objects with varying key order breaks the prefix. Use a stable serializer.
• Low traffic per prefix. If each customer has a custom prompt and sends one request every ten minutes, a five-minute TTL expires between every call. You pay the write premium every time.

A layout that caches well

Order the prompt from most stable to least stable:

1. Global instructions and policies, identical for everyone.
2. Tool definitions, sorted by name.
3. Long reference material shared by a tenant or workspace.
4. User profile and session details.
5. The conversation, newest turn last.

Put the cache boundary after the last block that is shared by enough traffic to stay warm. For a multi-tenant product that is often after block 2, not block 3.

Measure before you celebrate

Most APIs return how many input tokens were written to and read from the cache on each response. Log both. Plot the hit rate per prompt template, per hour. The first time you do this you will probably find one template stuck near zero because of a date string someone added for debugging. Fixing that one line is often the biggest cost reduction of the quarter.