Evals That Catch Regressions: Building a Golden Set Your Team Will Actually Maintain

Every team we meet has the same eval story: someone built a notebook with 30 prompts, it caught a bug once, and now it hasn't been run in four months. Then a model swap or a prompt tweak ships, support tickets spike, and nobody can tell which change broke what. The problem isn't that evals are hard — it's that most eval suites are designed to impress a stakeholder, not to survive a sprint.

This post is the pattern we actually use when we build LLM features for clients: a small, boring, high-signal golden set wired into CI, with scoring that an engineer can debug in under five minutes.

What a golden set actually is

A golden set is a frozen collection of inputs paired with either reference outputs or reference behaviors. It is not your entire test corpus. It is the subset you've decided represents the behaviors you refuse to regress on.

Think of it the way you'd think of a smoke test suite for a payments API: small enough to run on every PR, broad enough that if it passes you're probably fine.

A few properties that matter:

• Stable. You don't rewrite the inputs when the prompt changes. If an input stops being relevant, you archive it, you don't edit it in place.
• Diverse by failure mode, not by topic. Ten variations of "summarize this email" is not diversity. One summarization, one refusal case, one ambiguous instruction, one adversarial input, one long-context case — that's diversity.
• Owned. Every entry has a author and a reason-for-existing comment. If nobody remembers why a case is in the set, delete it.

Sizing it

For a single feature (say, a RAG-backed support assistant), we usually land somewhere between 40 and 150 cases. Below 40 you're not covering enough failure modes; above 150 and engineers start skipping runs because of latency and cost.

If your product has multiple surfaces — a chat UI, a batch classifier, an agentic workflow — each gets its own golden set. Don't mix them. The scoring criteria are different and the signal gets muddy.

Structuring cases so they survive contact with reality

The schema matters more than people think. If each case is just {input, expected_output}, you'll end up with brittle string-match assertions that break every time the model phrases things differently.

Here's the shape we use:

{
  "id": "rag-refund-policy-001",
  "category": "policy_lookup",
  "failure_mode": "hallucinated_policy",
  "input": {
    "user_message": "Can I get a refund after 60 days?",
    "context_docs": ["policy_v4.md#refunds"]
  },
  "assertions": [
    {"type": "must_contain_any", "values": ["30 day", "30-day", "thirty day"]},
    {"type": "must_not_contain", "values": ["60 day", "flexible", "case by case"]},
    {"type": "must_cite", "source": "policy_v4.md"}
  ],
  "notes": "Added after incident 2026-03-11 where model invented a 60-day grace period.",
  "owner": "maria"
}

Three things worth noting:

1. Multiple assertion types per case. Substring checks, forbidden phrases, citation requirements, and (sparingly) LLM-as-judge for semantic claims.
2. A failure_mode tag. When your dashboard shows "3 regressions in hallucinated_policy," that's instantly actionable. "3 regressions in refund questions" is not.
3. An incident link in notes. Every case tied to a real bug is a case nobody will argue about deleting.

Scoring without lying to yourself

There are three scoring approaches and you'll end up using all of them.

Deterministic checks

Substring, regex, JSON schema validation, citation presence, latency bounds, token count. These are cheap, fast, and unambiguous. Lean on them wherever you can. If your task has a structured output, schema validation alone will catch a surprising fraction of regressions.

LLM-as-judge

Use this for semantic correctness, tone, and groundedness — things you genuinely can't check with regex. A few rules we've learned the hard way:

• Use a different model family than the one under test. If you're evaluating a Claude-based pipeline, judge with GPT or Gemini. Models have correlated blind spots; vendor docs for both Anthropic and OpenAI acknowledge this when they discuss evaluation methodology.
• Make the judge output structured JSON with a reason field. You need the reason to debug false positives, which there will be many of.
• Calibrate against human labels on a sample. If your judge disagrees with humans more than ~15% of the time, your rubric is too vague.
• Judge one dimension per call. Asking a model to score helpfulness, accuracy, and tone in one shot gives mushier results than three separate calls, in our experience.

Human review

For the top of the funnel — new features, major prompt changes — nothing replaces a human going through 20 outputs. Build it into the ritual. We require a human review of a random 10-case sample before any prompt change ships to production, regardless of what the automated evals say.

Wiring it into CI without destroying your velocity

The trap: engineers run the full eval suite on every PR, it costs $4 and takes 90 seconds, and within two weeks they're disabling it "just for this one quick fix."

What works better is a tiered setup:

# .github/workflows/llm-evals.yml (simplified)
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/llm/**'
      - 'evals/**'

jobs:
  smoke:
    # ~15 cases, deterministic only, runs on every relevant PR
    runs-on: ubuntu-latest
    steps:
      - run: pnpm eval:smoke

  full:
    # Full golden set, including judge calls
    # Only on merge to main, or when PR has 'eval-full' label
    if: contains(github.event.pull_request.labels.*.name, 'eval-full')
    runs-on: ubuntu-latest
    steps:
      - run: pnpm eval:full

The smoke tier is your gate — fast, cheap, deterministic. The full tier is your safety net, run on merge and nightly against production traffic samples.

Store results, not just pass/fail

Every run should persist per-case outputs to object storage with the commit SHA, model ID, and prompt hash. When something regresses six weeks later, you want to diff against the last known good run, not re-infer what the previous output was.

The maintenance problem (which is the real problem)

Everyone talks about building evals. Nobody talks about maintaining them, which is why most eval suites rot.

The habits that keep ours alive:

• Every production incident adds a case. No exceptions. The PR that fixes the bug also adds the eval. This is the single highest-ROI rule.
• Monthly pruning. 30 minutes, one engineer, delete cases that no longer reflect the product or that haven't failed in a year and don't cover a unique failure mode.
• Owner rotation. Each category has a named owner for the quarter. Not a committee.
• Budget alarms. Track eval spend separately from production LLM spend. If evals cost more than ~5% of your production LLM bill, something is wrong — usually too many judge calls on cases that could be deterministic.

When to pay for a platform

Tools like Braintrust, Langfuse, Humanloop, and Promptfoo all solve pieces of this. They're worth it once you have more than two or three people editing prompts, or when non-engineers need to inspect runs.

Below that threshold, a repo with JSON files, a Python runner, and a Postgres table of results works fine and keeps you honest about what you're actually measuring. Don't buy infrastructure to avoid defining your rubric — the rubric is the hard part, and no vendor will write it for you.

Where we'd start

If you're staring at zero evals right now, do this in order over one week:

1. Pull 50 real production transcripts. Not synthetic ones.
2. Label each with a failure mode (even "no failure" is a label).
3. Pick the top three failure modes by frequency × severity. Write 10–15 golden cases per mode, using the schema above.
4. Wire the deterministic subset into CI as a required check on any prompt or model change.
5. Add one rule to your incident process: fixing a bug means adding a case.

That's it. You now have something small enough to maintain and sharp enough to catch the regressions that matter. Everything else — judge calibration, dashboards, drift detection — is an optimization you can earn your way into once the foundation holds.

If you want help building this into an existing product, that's a chunk of what we do on the AI engineering side — happy to compare notes.