Prompt caching math: when it saves 75% and when it costs you more
Cached prefixes are cheap to read and slightly expensive to write. We work out the break-even hit rate, the traps that keep it at zero, and a prompt layout that helps.
Infrastructure and cost
Tobi Adeyemi
Tobi ran inference infrastructure for a mid-sized SaaS company and now writes about tokens, caches, latency and invoices. He keeps a spreadsheet for everything and will send it to you if asked.
Stories by Tobi
The real cost of long context: a worked example at 40,000 calls a day
Stuffing 120,000 tokens into every request is easy to build and expensive to run. We price it, time it and compare it against a 6,000-token retrieval setup.