Postmortem: why our RAG answers drifted when nobody deployed anything
Over six weeks an internal help assistant got quietly worse with no model or prompt change. The causes were all in the retrieval layer, and all preventable.
Topic · 3 stories
Field reports
Hands-on tests and postmortems from real setups, with the numbers we measured and the mistakes we made.
Field test: three small models, two laptops and 200 invoices
We ran 3B, 7B and 14B open-weight models on two ordinary laptops to extract invoice fields. Speed, accuracy, memory and battery, measured rather than guessed.
What an eval suite actually catches (and the three things it never will)
A 240-case eval suite for a ticket summarizer, a cheaper model swap, and an honest accounting of which failures the suite caught and which reached users.