Eval Harnesses That Catch Regressions Before Users Do

Every team we talk to has the same story: they tweak a prompt on Friday, ship it, and by Tuesday support is asking why the assistant started refusing valid requests. The fix isn't more prompt engineering — it's an eval harness that runs before the change lands. Here's the setup we actually use on client projects, minus the theatre.

Why prompt regressions are worse than code regressions

A broken function throws. A broken prompt keeps answering — just worse. It might get 3% more refusals, drop citations in half its RAG answers, or start hallucinating currency codes. Nothing crashes. Your dashboards stay green. The only signal is a slow drip of user complaints two weeks later.

Standard unit tests don't help because LLM outputs are non-deterministic and often free-form. You need a harness built around three things: a curated dataset, graders that tolerate variation, and a CI gate that blocks merges when quality drops.

The three failure modes we keep seeing

• Prompt drift: someone edits the system prompt to fix one edge case and silently breaks five others.
• Model swaps: switching from gpt-4o to gpt-4.1-mini to save cost, or from Claude 3.5 Sonnet to a newer Claude version, without checking behaviour on your actual tasks.
• Retrieval drift: the embeddings model changes, the chunking changes, or the corpus grows, and top-k results quietly degrade.

An eval harness catches all three with the same machinery.

Start with a dataset that reflects real traffic

Synthetic evals feel productive and prove almost nothing. Before writing a single grader, pull 100–300 real (or realistic) examples from production logs. Strip PII, then bucket them:

• Golden path: the top 10–15 request shapes that make up most of your volume.
• Known-hard: cases that historically failed or got escalated.
• Adversarial: prompt injection attempts, ambiguous inputs, wrong-language queries.
• Should-refuse: things the model should decline (out-of-scope, unsafe).

We've found that 150 well-chosen examples beat 5,000 synthetic ones. Every example gets tagged with a bucket and an expected outcome — not necessarily a full expected answer, but at minimum a set of assertions.

{
  "id": "inv-refund-042",
  "bucket": "golden_path",
  "input": {
    "user": "Can I get a refund on order #A-3319? It arrived cracked.",
    "context": { "order_id": "A-3319", "status": "delivered" }
  },
  "assertions": [
    { "type": "tool_called", "name": "create_refund_request" },
    { "type": "contains_field", "path": "tool_args.order_id", "value": "A-3319" },
    { "type": "tone", "expect": "empathetic" },
    { "type": "no_promise", "pattern": "guaranteed|definitely will" }
  ]
}

Notice there's no single "correct answer" string. That's the point.

Graders: pick the cheapest one that works

Graders fall on a spectrum from deterministic to judgemental. Use the cheapest tier that gives a reliable signal for each assertion.

Tier 1: deterministic checks

These are free, fast, and non-flaky. Use them for anything you can:

• Was the correct tool called with the correct arguments?
• Does the JSON parse and match the schema?
• Does the output contain (or not contain) specific tokens, URLs, currencies?
• Is the latency and token count within budget?

Roughly 60–70% of our assertions on a typical project are tier 1. If you find yourself reaching for an LLM judge to check whether the model called create_refund_request, you've overcomplicated it.

Tier 2: embedding similarity

For "did the answer cover roughly the right ground" questions, embed the reference answer and the actual answer, cosine-compare, and threshold. It's noisier than it looks — semantic similarity of 0.82 might mean anything — so we mostly use it as a regression signal (this run vs last run) rather than an absolute bar.

Tier 3: LLM-as-judge

Reserve this for genuinely qualitative properties: tone, faithfulness to sources, refusal appropriateness. A few rules that saved us pain:

• Use a stronger model as judge than as generator. Judging with the same model you're testing is a known bias trap; both OpenAI and Anthropic call this out in their eval guidance.
• Score with a rubric, not a vibe. Ask for a 1–5 on named criteria with short justifications, not "is this good?"
• Pin the judge model version. Rotating the judge is how you get mysterious quality shifts that aren't real.
• Sample-audit the judge. Once a month, a human reviews 20 judgements. If the judge and human disagree more than ~15% of the time, the rubric needs work.

Wire it into CI, not into a notebook

A harness that only runs when someone remembers to run it is not a harness. Ours runs in three places:

1. Pre-merge on PRs that touch prompts, models, retrieval config, or tool schemas. Blocking gate on the golden-path and should-refuse buckets.
2. Nightly on the full dataset. Non-blocking, but posts a diff to Slack.
3. On-demand for model comparisons — the same dataset run across GPT, Claude, and Gemini before we recommend a switch.

A minimal runner looks like this:

import asyncio, json
from statistics import mean

async def run_case(case, generator, graders):
    output = await generator(case["input"])
    results = [g(case, output) for g in graders]
    return {
        "id": case["id"],
        "bucket": case["bucket"],
        "passed": all(r["passed"] for r in results),
        "results": results,
        "cost_usd": output.cost,
        "latency_ms": output.latency_ms,
    }

async def run_suite(dataset, generator, graders, concurrency=8):
    sem = asyncio.Semaphore(concurrency)
    async def guarded(c):
        async with sem:
            return await run_case(c, generator, graders)
    return await asyncio.gather(*[guarded(c) for c in dataset])

def summarise(runs):
    by_bucket = {}
    for r in runs:
        by_bucket.setdefault(r["bucket"], []).append(r["passed"])
    return {b: mean(v) for b, v in by_bucket.items()}

The CI job compares the summary against a stored baseline (main's last green run) and fails if any bucket drops by more than a configured tolerance — usually 3 percentage points for golden-path, 5 for known-hard.

Store every run

Dump full inputs, outputs, grader results, token counts, and costs to durable storage — object store is fine. When a PR fails the gate, you want to open a single link and diff the actual outputs against main, not re-run and hope.

Cost guardrails so evals don't eat your budget

A 300-case suite that hits a frontier model on every PR gets expensive fast. Some things that keep it reasonable:

• Two tiers of dataset: a 40-case "smoke" suite that runs on every PR, and the full 300-case suite nightly and on prompt/model changes.
• Cache generator outputs by input hash when the model and prompt haven't changed — most PRs don't touch the AI code path at all.
• Use prompt caching on providers that support it (Anthropic's prompt caching and OpenAI's automatic prompt caching both apply to eval runs with shared system prompts). The system prompt is identical across every case; that's free savings.
• Skip tier-3 judges on the smoke suite. Deterministic and embedding checks only.

In our experience, a well-tuned harness costs a few dollars per PR run and tens of dollars per nightly full run. If yours costs more, the dataset is probably too big or you're judging things that should be deterministic.

What actually breaks (and how the harness catches it)

A few war-story patterns from real projects:

• A prompt tweak to reduce verbosity dropped tool-call rates by 11% on the golden path. Caught by tier-1 tool_called assertions on the PR.
• Switching to a cheaper embeddings model cut retrieval quality on domain jargon. Caught by faithfulness judge scores dropping on the RAG bucket, even though top-line accuracy looked fine.
• A new model version got more cautious and started refusing 8% of legitimate requests. Caught by the should-refuse bucket's false-positive rate spiking.

None of these would have shown up in manual QA before the next sprint.

Where we'd start

If you have no evals today, don't try to build the perfect harness. In one week you can:

1. Day 1–2: pull 50 real examples, bucket them, write assertions.
2. Day 3: write tier-1 deterministic graders. Get a pass/fail number.
3. Day 4: wire it to CI as a non-blocking check. Watch it for a few PRs.
4. Day 5: flip it to blocking on the golden-path bucket only. Add tier-3 judges for the one or two qualities that actually matter.

Then grow the dataset every time something breaks in production — each incident becomes a permanent test case. That's the compounding part. If you want a hand designing the harness around a specific product, that's the kind of work we do in our AI engagements.