LLM-as-judge: a regression caught, a regression fixed

Why this matters

Type-checks and unit tests don’t catch chat-behavior regressions. A test that asserts expect(response).toContain("Scripps") passes whether the answer is grounded in real content or invented out of the embedding void. The only way to know if a chatbot still behaves correctly after a change is to run real questions through it and score the answers.

On peterwd.com the eval suite is 32 curated questions covering grounded answers, refusals, tool-call selection, prompt injection, and out-of-scope behavior. Each answer is scored on four axes — groundedness, voice, citation alignment, behavior correctness— by a separate model acting as judge. Per-run cost is about $0.35, wall-clock about 6 minutes.

What broke

The chat route added a check: when retrieval came back thin (top similarity below 0.45), strip citations from the response. The intent was honest — if the model isn’t really using the chunks, don’t pretend they’re sources.

The threshold was set by intuition, not by sampling. After shipping, the eval flagged four happy-path questions all scoring ~0.30. The pattern was identical — well-written, grounded, factually correct answers with the judge complaint: “no citations returned despite shouldGround=true.”

What the judge said

Excerpt from the judge’s structured output on peterwd-identity— the kind of feedback that drives a fix:

“The answer contains rich, specific biographical claims (Senior Enterprise Architect at Scripps Health, Frenti, coding since 1986, etc.) that should be grounded in retrieved content from about.mdx, but zero citations are returned — a critical failure for a regression test explicitly designed to verify the Identity section was added. Voice is strong: direct, concrete, no banned words. However, the missing citations and lack of retrieval mean the chatbot failed its core task despite producing a plausible-sounding answer.”

The fix

Lowered the citation-suppress threshold and decoupled it from the model-prompt “low-confidence” flag. The fallback prompt that asks the model to suggest adjacent sections still fires at 0.45; only the citation header is gated, and only on near-empty retrieval.

  // Skip citations when retrieval was thin
- const responseCitations = lowConfidence ? [] : citations;
+ const noCitations = matched.length === 0 || topSim < 0.25;
+ const responseCitations = noCitations ? [] : citations;

The change is small. The discipline isn’t. The eval is what made the sub-threshold choice visible — without it, the regression would have shipped and been discovered by a visitor.

Before / after

Six questions changed status across the two runs. Four were the regression. Two were previously parked as pending— they were waiting on new content (a per-employer Scripps Health section in about.mdx) that landed in the same commit batch.

Four example questions, four behaviors

A representative slice from today’s suite. Each card shows the question, the chatbot’s answer, what was retrieved, the judge’s overall score, and the judge’s reasoning — the same structure that lands in evals/results/*.md after every run.

What an LLM-judge eval suite buys you

For client projects shipping AI features, this is the discipline that closes the gap between “it works on my machine” and “it works in production.”

• Regressions surface in minutes. Run the suite on every PR that touches the model, the prompt, or the retrieval pipeline. The 30/30 baseline is the contract.
• Judge reasoning is the diff comment. When a score drops, the judge explains why — not “assertion failed at line 47.” That qualitative signal is what makes the fix obvious.
• Pending tests document missing content. The two questions about Scripps Health were flagged pending until the content existed. When the content shipped, the tests un-paused and validated themselves.
• Cheap insurance. $0.35 per run. Six minutes wall-clock. A single shipped regression that wastes a sales conversation costs more.