Topic · 4 stories
Token math, caching, time to first token and the budgets that decide whether a feature survives its first invoice.
Users judge a chat feature in the first second and a half. How to split that budget across network, guardrails, retrieval and the model, and what to show while they wait.
Cached prefixes are cheap to read and slightly expensive to write. We work out the break-even hit rate, the traps that keep it at zero, and a prompt layout that helps.
We ran 3B, 7B and 14B open-weight models on two ordinary laptops to extract invoice fields. Speed, accuracy, memory and battery, measured rather than guessed.
Stuffing 120,000 tokens into every request is easy to build and expensive to run. We price it, time it and compare it against a 6,000-token retrieval setup.