Agents
Agents need an undo button before they need more autonomy
An agent that is right 97% of the time still makes dozens of wrong moves a day. Reversibility, not accuracy, makes that tolerable. An argument, with a pattern.
Independent · Applied AI · Since 2024
We read the model card so you don't have to (you should)
A publication for people who have argued with a JSON parser at 1 a.m. Expect cost spreadsheets, honest field tests and the occasional confession about a prompt we're not proud of.
Latest from the desk
Product and design
Latency budgets for chat interfaces: spend 1.5 seconds on purpose
Users judge a chat feature in the first second and a half. How to split that budget across network, guardrails, retrieval and the model, and what to show while they wait.
Cost and latency
Prompt caching math: when it saves 75% and when it costs you more
Cached prefixes are cheap to read and slightly expensive to write. We work out the break-even hit rate, the traps that keep it at zero, and a prompt layout that helps.
Retrieval and RAG
Postmortem: why our RAG answers drifted when nobody deployed anything
Over six weeks an internal help assistant got quietly worse with no model or prompt change. The causes were all in the retrieval layer, and all preventable.
Small and on-device
How to read a model card skeptically, line by line
Model cards are marketing and documentation at once. A practical checklist for separating the claims you can rely on from the ones you need to test yourself.
Small and on-device
Field test: three small models, two laptops and 200 invoices
We ran 3B, 7B and 14B open-weight models on two ordinary laptops to extract invoice fields. Speed, accuracy, memory and battery, measured rather than guessed.
Cost and latency
The real cost of long context: a worked example at 40,000 calls a day
Stuffing 120,000 tokens into every request is easy to build and expensive to run. We price it, time it and compare it against a 6,000-token retrieval setup.
Get the next issue before your next sprint
One essay every other Thursday on evals, retrieval, agents, latency or cost, with the arithmetic shown and the code attached.
Free · No sponsored posts · 18,400 readers