Benchlight

Benchlight lets you write test cases for prompts the way you write unit tests: inputs, expected properties and graders. Run the suite against several models or prompt versions and get a table of pass rates, costs and latency, with diffs for every case that changed.

Teams shipping LLM features use it in CI to catch regressions when they edit a prompt or switch providers. Graders can be exact matches, regular expressions, code or another model with a rubric. Model-graded checks inherit the grader's blind spots, so Benchlight recommends keeping a human-reviewed sample, and its web viewer is basic compared with commercial evaluation platforms.