What an eval suite actually catches (and the three things it never will)

Last spring a team I advise wanted to swap the model behind their support-ticket summarizer for one that cost about 60% less per call. They had an eval suite: 240 real tickets, each with a reference summary and a handful of checks. The cheaper model scored 231 out of 240. The incumbent scored 236. Five cases apart, the swap looked safe.

It mostly was. But the interesting part is what those 240 cases were good at, and what they had no way to see.

What the suite caught

Grouping the failures by kind made the picture clearer than the headline score did. Of the nine cases the cheaper model failed, the suite flagged:

• Four format breaks. The summary was supposed to be JSON with issue, customer_tone and next_step. The new model occasionally wrapped it in a Markdown code fence. A parser assertion caught every one.
• Three dropped facts. Tickets with an order number in the last paragraph lost it in the summary. A simple “must contain” check on extracted IDs found them.
• Two tone misreads. An angry customer labelled “neutral”. A grader model compared the label to a human-written one and disagreed.

These are the failures eval suites are built for: deterministic, checkable, and repeated across many inputs. If your suite does nothing else, it should catch these before a model, prompt or temperature change goes out.

def check(case, output):
    results = {}
    try:
        data = json.loads(output)
        results["valid_json"] = True
    except json.JSONDecodeError:
        return {"valid_json": False}
    results["keeps_ids"] = all(i in data["issue"] for i in case.order_ids)
    results["tone_match"] = data["customer_tone"] == case.expected_tone
    return results

The three things it never will

The swap shipped. Within two weeks, support leads raised three problems the suite could not have caught.

Distribution shift. The 240 tickets were sampled in January. In March the company launched a subscription tier, and a new class of billing tickets appeared. Neither model had been tested on them. The suite measured last quarter’s traffic with great precision.

Preference, not correctness. Agents found the new summaries technically complete but harder to scan. Sentences were longer; the next step was buried. No assertion in the suite measured “I can act on this in three seconds”, and a grader model, asked whether the summary was accurate, happily said yes.

Rare and costly cases. One ticket in roughly 4,000 mentions a legal threat. The suite contained zero of them. The new model summarized one as “customer unhappy with refund timeline”. Nobody would have found that in 240 samples, because at that rate you need several thousand to expect even one.

How we changed the suite

None of this means the suite was a mistake. It means a suite is a regression detector, not a quality guarantee. After the review, the team made four changes:

The last row matters most. Edits are free labels. Every time a support agent rewrites a summary, that ticket becomes a candidate eval case, already annotated with what a human wanted instead.

A rule of thumb

If a failure can be stated as an assertion, it belongs in the suite and should block a release. If it can only be stated as a preference, it belongs in a small, recurring human review. If it is rare, you will not find it by sampling; you have to write it on purpose. Most teams have the first kind, skip the second, and discover the third from a support escalation.