Independent · Applied AI · Since 2024
Dispatches from the inference room, set in hot type
A weekly bulletin for practitioners, printed the old way: one topic, worked by hand, checked twice. Tokens counted, footnotes kept, no telegrams about the next big model.
Browse by topic
9
stories so far, by 4 writers
Latest from the desk
Prompt caching math: when it saves 75% and when it costs you more
Cached prefixes are cheap to read and slightly expensive to write. We work out the break-even hit rate, the traps that keep it at zero, and a prompt layout that helps.
Postmortem: why our RAG answers drifted when nobody deployed anything
Over six weeks an internal help assistant got quietly worse with no model or prompt change. The causes were all in the retrieval layer, and all preventable.
How to read a model card skeptically, line by line
Model cards are marketing and documentation at once. A practical checklist for separating the claims you can rely on from the ones you need to test yourself.
Field test: three small models, two laptops and 200 invoices
We ran 3B, 7B and 14B open-weight models on two ordinary laptops to extract invoice fields. Speed, accuracy, memory and battery, measured rather than guessed.
The real cost of long context: a worked example at 40,000 calls a day
Stuffing 120,000 tokens into every request is easy to build and expensive to run. We price it, time it and compare it against a 6,000-token retrieval setup.
Browse by topic
- Cost and latency4Token math, caching, time to first token and the budgets that decide whether a feature survives its first invoice.Browse
- Evals3Test suites, graders and regression checks for model-backed features, and what each kind of check can and cannot catch.Browse
- Agents3Tool-using systems that take actions, and the guardrails, undo paths and review steps they need.Browse
- Product and design3Interfaces, expectations and feedback loops for AI features people use every day.Browse
- Field reports3Hands-on tests and postmortems from real setups, with the numbers we measured and the mistakes we made.Browse
- Retrieval and RAG2Chunking, embeddings, indexes and the quiet ways retrieval-augmented answers change over time.Browse
Get the next issue before your next sprint
One essay every other Thursday on evals, retrieval, agents, latency or cost, with the arithmetic shown and the code attached.
Free · No sponsored posts · 18,400 readers