What an eval suite actually catches (and the three things it never will)
A 240-case eval suite for a ticket summarizer, a cheaper model swap, and an honest accounting of which failures the suite caught and which reached users.
Applied AI PublicationOctober 28, 20269 stories · 7 topics
Field notes from the unglamorous middle of AI products
No launch coverage, no leaderboards. Essays, postmortems and interviews about what happens after a model meets real users, real budgets and real Monday mornings.
In brief
Q&A: designing AI features that users forgive when they are wrong
A design lead on why her team stopped chasing accuracy alone, the edit-rate metric they track instead, and the small interface changes that halved complaints.
Editors' picks
AgentsAgents need an undo button before they need more autonomy
An agent that is right 97% of the time still makes dozens of wrong moves a day. Reversibility, not accuracy, makes that tolerable. An argument, with a pattern.
Latest from the desk
Retrieval and RAG
Postmortem: why our RAG answers drifted when nobody deployed anything
Over six weeks an internal help assistant got quietly worse with no model or prompt change. The causes were all in the retrieval layer, and all preventable.
Small and on-device
How to read a model card skeptically, line by line
Model cards are marketing and documentation at once. A practical checklist for separating the claims you can rely on from the ones you need to test yourself.
Small and on-device
Field test: three small models, two laptops and 200 invoices
We ran 3B, 7B and 14B open-weight models on two ordinary laptops to extract invoice fields. Speed, accuracy, memory and battery, measured rather than guessed.
Browse by topic
- 01EvalsTest suites, graders and regression checks for model-backed features, and what each kind of check can and cannot catch.3 stories
- 02Cost and latencyToken math, caching, time to first token and the budgets that decide whether a feature survives its first invoice.4 stories
- 03Retrieval and RAGChunking, embeddings, indexes and the quiet ways retrieval-augmented answers change over time.2 stories
- 04AgentsTool-using systems that take actions, and the guardrails, undo paths and review steps they need.3 stories
- 05Product and designInterfaces, expectations and feedback loops for AI features people use every day.3 stories
- 06Small and on-deviceSmall and open-weight models on laptops and phones: quantization, memory and honest benchmarks.2 stories
- 07Field reportsHands-on tests and postmortems from real setups, with the numbers we measured and the mistakes we made.3 stories
Get the next issue before your next sprint
One essay every other Thursday on evals, retrieval, agents, latency or cost, with the arithmetic shown and the code attached.
Free · No sponsored posts · 18,400 readers