Eval Harnesses That Actually Catch Regressions: A Practical Setup

Most LLM eval suites we inherit from clients tell the team nothing useful. They pass when the model quietly gets worse, they fail loudly on cosmetic changes, and nobody trusts the number at the bottom. If your evals aren't gating deploys, they're theatre.

This is the harness structure we've settled on after enough late-night rollbacks. It's opinionated, cheap to run, and — more importantly — it actually catches regressions when you swap models, tweak a prompt, or refactor a retriever.

What a regression actually looks like

Before designing the harness, get honest about what you're trying to catch. In our experience, LLM regressions fall into four buckets:

1. Silent quality drops. The model still answers, but hallucinates a field, drops a citation, or gets subtly less specific.
2. Format breakage. JSON that used to parse now includes a stray markdown fence, or a tool call arguments schema shifts.
3. Behavioural drift. Refusals appear where they didn't before, tone changes, or the model starts hedging on questions it used to answer directly.
4. Latency and cost regressions. Same output quality, but 3x the tokens or twice the p95 latency.

A good harness needs a distinct signal for each. One aggregate "quality score" hides all four.

The four-layer harness

We structure evals in four layers, run in order, cheapest to most expensive. If a layer fails, later layers still run (you want the full picture), but the CI gate short-circuits based on which layer failed and how badly.

Layer 1: Deterministic checks

These are pure code. No model calls. They cover schema validation, required fields, forbidden strings, citation presence, length bounds, and anything else you can express as an assertion.

from pydantic import BaseModel, ValidationError

class SupportReply(BaseModel):
    answer: str
    citations: list[str]
    escalate: bool

def check_deterministic(raw: str, case: dict) -> dict:
    failures = []
    try:
        parsed = SupportReply.model_validate_json(raw)
    except ValidationError as e:
        return {"pass": False, "failures": [f"schema: {e.errors()[0]['msg']}"]}

    if case.get("requires_citation") and not parsed.citations:
        failures.append("missing_citation")
    if len(parsed.answer) > 1200:
        failures.append("answer_too_long")
    for banned in case.get("banned_phrases", []):
        if banned.lower() in parsed.answer.lower():
            failures.append(f"banned_phrase:{banned}")

    return {"pass": not failures, "failures": failures}

Deterministic checks are boring and non-negotiable. They run in milliseconds and catch the majority of "the deploy broke" incidents. If your harness doesn't have this layer, add it before anything else.

Layer 2: Reference-based scoring

For cases where you have a known-good answer — extractions, classifications, structured summarisation — use exact match, F1, or set overlap on the fields that matter. Not on the whole string.

For a support-ticket classifier, we score the category field with exact match and the tags field with Jaccard overlap. We don't score the free-text explanation with anything reference-based; that's Layer 3's job.

Layer 3: LLM-as-judge, with structure

This is where most teams go wrong. They ask a strong model "is this answer good, 1-10?" and treat the number as truth. That produces noisy, drifting scores that correlate weakly with anything users care about.

A judge worth trusting has three properties:

• Rubric-driven. The judge scores specific dimensions (faithfulness, completeness, tone) against a written rubric, not a vibe.
• Pairwise where possible. Ask "is A better than B on dimension X?" rather than absolute scores. Pairwise comparisons are far more stable across judge model versions (Anthropic's guidance on evals and OpenAI's evals cookbook both push this).
• Calibrated against humans. Before you trust the judge, hand-label ~50 cases and measure judge agreement with humans. If it's below ~80%, tighten the rubric or change the judge model.

JUDGE_PROMPT = """You are grading a support agent reply against a rubric.

RUBRIC:
- faithfulness: does every claim in the reply appear in the provided context? (yes/no)
- completeness: does the reply address every part of the user's question? (yes/partial/no)
- tone: is the reply professional and non-condescending? (yes/no)

Return JSON: {"faithfulness": "...", "completeness": "...", "tone": "...", "notes": "..."}

CONTEXT:
{context}

USER QUESTION:
{question}

AGENT REPLY:
{reply}
"""

Run the judge with temperature=0 and a structured output schema. Use a different model family from the one you're evaluating when you can — grading Claude Sonnet with GPT-4.1 (or vice versa) reduces the risk of a model preferring its own outputs. That self-preference bias is well-documented and shows up in practice.

Layer 4: Behavioural probes

A small, curated set (30-100 cases) of edge behaviours: prompt injections, adversarial user turns, questions that should trigger refusals, and questions that should not. These aren't graded on quality — they're graded on whether the model did the right categorical thing.

We keep this set frozen and version-controlled. Every model swap runs it. When Gemini, Claude, or GPT ship a new snapshot, this is the layer that tells us whether the safety and refusal behaviour shifted underneath us.

The golden dataset problem

Your harness is only as good as the cases in it. Two rules we enforce:

Rule 1: Golden cases come from production. Not from the PM's imagination. We sample real traffic weekly, strip PII, and have a domain expert label a slice. Synthetic cases are fine for coverage of rare paths, but the core set must reflect what users actually send.

Rule 2: Split into tiers.

• Smoke set (20-50 cases): runs on every PR, under 60 seconds.
• Full set (300-1000 cases): runs nightly and pre-deploy.
• Stress set (behavioural probes + long-tail): runs weekly and before any model swap.

The smoke set is the one engineers actually feel. If it takes more than a couple of minutes, they'll start skipping it or merging around it.

Wiring it into CI without lying

The gate has to be honest. Two failure modes we've seen:

1. Flaky pass thresholds. "Fail if score < 0.85" — but the score is noisy ±0.04 across runs, so half the failures are noise. Fix: run smoke set with temperature=0, seed where the API supports it, and compare against a rolling baseline rather than an absolute number.
2. Aggregate scores hiding regressions. A 2% overall drop can mean one critical category collapsed. We break scores out by tag (billing, auth, refunds) and gate per-tag, not just on the total.

Our GitHub Actions setup, simplified:

- name: Run smoke evals
  run: python -m evals.run --suite smoke --baseline main
- name: Gate on regressions
  run: python -m evals.gate \
    --max-tag-regression 0.05 \
    --max-overall-regression 0.02 \
    --require-no-new-schema-failures

The gate compares against the last green run on main, not a hardcoded number. When quality genuinely improves, the baseline moves with it.

Cost guardrails inside the harness

Every eval run logs tokens in, tokens out, and wall-clock latency per case. We fail the build if median tokens-per-case rises more than ~15% without an explicit override in the PR description. This has caught more than one "I just added a few more examples to the prompt" change that would have doubled our monthly bill.

For the judge itself: judge calls dominate eval cost. Batch them, cache them keyed on (prompt_hash, output_hash), and use the cheaper tier of your judge model family where the rubric is simple. A judge that costs more per run than the feature it grades will get turned off.

What breaks this setup

Honest limitations:

• Judges drift when vendors ship new snapshots. Pin judge model versions explicitly and re-calibrate against humans quarterly.
• Golden sets rot. Product changes, user behaviour changes, and last quarter's cases stop being representative. Budget an afternoon a month for curation.
• Multi-turn and agent evals are harder. This harness handles single-turn well. For agent trajectories, you need trace-level scoring — that's a separate post.

Where we'd start

If you're staring at a codebase with no evals and a nervous product manager: don't build the whole four-layer thing on day one. Start here.

1. Pull 50 real production requests. Label them by hand.
2. Write Layer 1 (deterministic) checks for schema and required fields. Run in CI on every PR.
3. Add a simple rubric-driven judge for the top failure mode you actually see in support tickets. Calibrate against your 50 labels.
4. Gate merges on Layer 1 hard, Layer 3 as an advisory comment on the PR for two weeks, then flip it to blocking once the team trusts the score.

That's a week of work and it will catch more regressions than the elaborate dashboard you were planning. Everything else — behavioural probes, per-tag gates, cost guardrails — layers on once the foundation is boring and reliable. If you'd like help wiring this into your stack, our AI engineering team does this for a living.