Postmortem: why our RAG answers drifted when nobody deployed anything
Over six weeks an internal help assistant got quietly worse with no model or prompt change. The causes were all in the retrieval layer, and all preventable.
Topic · 3 stories
Evals
Test suites, graders and regression checks for model-backed features, and what each kind of check can and cannot catch.
How to read a model card skeptically, line by line
Model cards are marketing and documentation at once. A practical checklist for separating the claims you can rely on from the ones you need to test yourself.
What an eval suite actually catches (and the three things it never will)
A 240-case eval suite for a ticket summarizer, a cheaper model swap, and an honest accounting of which failures the suite caught and which reached users.