Infrastructure and cost
Tobi ran inference infrastructure for a mid-sized SaaS company and now writes about tokens, caches, latency and invoices. He keeps a spreadsheet for everything and will send it to you if asked.
Cached prefixes are cheap to read and slightly expensive to write. We work out the break-even hit rate, the traps that keep it at zero, and a prompt layout that helps.
Stuffing 120,000 tokens into every request is easy to build and expensive to run. We price it, time it and compare it against a 6,000-token retrieval setup.