Postmortem: why our RAG answers drifted when nobody deployed anything

This is a lightly anonymized postmortem from a company we’ll call Halyard, which runs an internal assistant that answers employee questions about policies, tools and benefits. The team gave permission to publish it on the condition that we keep the boring parts in. The boring parts are the lesson.

What happened

Over six weeks, the share of answers employees rated helpful fell from 82% to 69%. Nobody had changed the model, the prompt or the temperature. The deploy log for the assistant service was empty for the entire period. The first theory was that the model provider had changed something. It had not.

The real causes were three separate changes to the retrieval layer, each made by a different person for a reasonable reason.

Timeline

• Week 1. The docs team migrated the policy wiki to a new platform. Old pages were kept as redirects but also stayed in the export, so the nightly indexing job ingested both. The index grew from about 9,400 chunks to 15,100.
• Week 3. An engineer raised the chunk size from 512 to 1,024 tokens to reduce “answers missing context”. The change applied only to newly ingested documents. The index now held chunks of two sizes, which score differently against the same query.
• Week 4. The platform team upgraded the embedding model to a newer version and re-embedded new documents. Older chunks kept vectors from the previous model. Similarity scores between the two sets were not comparable, and nobody noticed because search still returned results.
• Week 6. An employee reported that the assistant quoted a parental leave policy replaced the previous year. The investigation began.

Why it was hard to see

Every answer looked plausible. The model was faithfully summarizing whatever it was given, and what it was given had changed. The team’s eval suite tested the generation step with fixed context passages, so it passed every night. Retrieval had no tests at all.

Logging made it worse. The service recorded the question and the answer but not which chunks were retrieved. Reconstructing what the model saw for a given complaint took a day per case.

What we changed

The fixes were mostly plumbing:

1. Pin an index version. Every answer now records the index build ID, embedding model and chunking config it used.
2. Log retrieved chunk IDs with every answer, along with their scores and source document dates.
3. Rebuild, never patch. Changes to chunking or embeddings trigger a full rebuild into a new index, evaluated before traffic switches.
4. Test retrieval directly. A set of 120 questions with known correct source documents, scored on recall at 5. It runs on every rebuild and blocks the switch if recall drops more than two points.
5. Deduplicate at ingest by canonical URL and content hash.

answer_log = {
    "question_id": qid,
    "index_build": "idx-2026-07-09-b",
    "embed_model": "embed-v3",
    "chunking": {"size": 512, "overlap": 64},
    "retrieved": [{"chunk": c.id, "score": round(c.score, 3), "doc_date": c.doc_date} for c in hits],
}

Results

After a full rebuild with deduplication, the index dropped back to about 9,600 chunks. Recall at 5 on the new retrieval test went from 0.71 on the drifted index to 0.90. Helpful ratings returned to 81% within two weeks.

The broader point is the one we keep relearning. A RAG system has two models and a database, and only one of those three was under test. If the retrieval layer can change without a deploy, it needs its own release process, its own evals and its own entry in the incident runbook.