Evaluating LLM features before you ship them
Most AI features fail in production because nobody built a way to tell whether a prompt change made things better or worse. Here is the evaluation harness we build first.
Key takeaways
- Build the evaluation harness before the feature, not after the first incident.
- Fifty to two hundred real, labelled inputs beat any volume of synthetic examples.
- Mix cheap deterministic graders with calibrated model graders, and block merges on regressions.
What does it mean to evaluate an LLM feature?
Evaluating an LLM feature means running every prompt, model or retrieval change against a fixed set of labelled real inputs and scoring the output with graders you trust, so a regression blocks the merge instead of reaching users. The harness is a test suite, not a dashboard.
The question every AI team has to be able to answer
The single biggest predictor of whether an AI feature survives contact with real users is not the model you picked. It is whether the team can answer one question: did that change make it better?
Without an evaluation harness, every prompt edit is a coin flip dressed up as engineering judgement. Someone tweaks the system prompt, three examples look better in a scratch file, and it ships. Two weeks later a different failure mode appears in a segment nobody was testing.
Start with a golden dataset of real inputs
We build the harness before the feature. It starts with a golden dataset — fifty to two hundred real inputs, labelled with what a correct response looks like. Not synthetic examples. Real ones, pulled from the actual queue the feature will serve.
Labelling is where the domain knowledge goes in. For a support assistant, that means the answer a senior agent would give, the documents it should cite, and the cases where the correct behaviour is to escalate rather than answer.
The two grader families, and what each one catches
| Grader type | Examples | Catches | Cost |
|---|---|---|---|
| Deterministic | Valid JSON, citation exists, token budget, latency | Structural and contract failures | Negligible |
| Model-graded | Faithfulness to context, tone, refusal correctness | Subtle quality regressions | Needs calibration and spend |
Run it in CI and let it block the merge
Every prompt or model change runs the full suite in CI. A regression on any grader blocks the merge. This sounds heavy. In practice it takes about two days to set up and it is the difference between an AI feature you can iterate on and one that everyone is afraid to touch.
The second-order effect matters more than the first. Once the suite exists, the team stops arguing about whether a change is an improvement and starts arguing about whether the graders are measuring the right thing — which is a much more productive argument.
- ≈2 days
- Setup time per feature
- 31
- Regressions caught pre-merge
- 4x
- Prompt changes shipped per week
FAQFAQ
Frequently asked questions
About the author
Ravi builds the applied AI systems we ship: retrieval pipelines, evaluation harnesses and the guardrails that keep a model useful once real users reach it. He writes about the engineering that decides whether an AI feature survives production.
- LLM evaluation
- Retrieval-augmented generation
- Prompt engineering
- AI cost modelling
