Two weeks before launch, someone asks the question that stops every AI feature meeting cold: how good is it? Not vibes-good. Numbers-good. And the honest answer is that no one has time to build the labelled dataset that would answer it properly. Building one right — collecting a representative sample, writing ground truth for each case, getting a second reviewer to check the first — takes longer than the feature took to build.
The two bad options everyone reaches for are building the dataset anyway and slipping the date, or shipping without any eval and finding out from support tickets. Both are avoidable. The dataset doesn't need to be large or representative to be useful; it needs to answer one specific question you actually have. That's a much smaller problem than the one most teams think they're solving.
We've built and shipped enough LLM features — vision-model takeoff scoring for BuildAEC, retrieval-backed chat over a vector store of roughly 21,000 embedded chunks — to know the eval that ships in a day looks nothing like the eval a research paper would run. It looks like a rubric, twenty adversarial cases, and a judge you don't fully trust yet. That's enough to make the ship decision responsibly.
Name the decision before you name the metric
Every eval question is really two questions stacked on top of each other: what am I deciding, and what evidence would change my mind? Skip straight to metrics and you end up measuring 'accuracy' in the abstract, which is expensive to define and almost never the thing blocking the ship. Name the decision first. Is it 'ship prompt v2 instead of v1'? 'Route anything under 80% confidence to a human'? 'Turn the feature on for all tenants instead of the five beta customers'? Each of those is a comparative question, not an absolute one, and comparative questions need far fewer cases to answer honestly.
This also settles a question that should come before any eval work: whether the feature needs an LLM at all. If the honest answer to 'how do you know it's right' is 'a database query would have told us for certain,' the decision about whether the feature needs an LLM in the first place is more urgent than the evaluation of the one you built.
Write the rubric before you write the scorer
A rubric is a list of specific, checkable failure modes for this feature, in this domain, written by someone who has looked at real output and knows what actually breaks. It is not 'is the answer good' — that question can't be scored consistently by two different people, let alone by an LLM. It is a set of yes/no or graded checks tied to the specific ways this feature fails.
On a construction takeoff feature, for instance, the rubric isn't 'is the quantity correct.' It's closer to this, because each failure mode has a different fix:
- Does the extracted quantity match the source drawing, within tolerance?
- Is the unit correct — square feet, not linear feet or count?
- Did the model invent a line item that doesn't exist anywhere in the source document?
- Does the confidence score on this line actually correlate with whether it needed correction?
That last criterion matters more than the first three combined once you've shipped, because it's what decides whether confidence scoring is actually making output reviewable or just producing a number nobody checks. We built exactly this kind of per-line confidence gating into BuildAEC's vision-model takeoff, and the rubric we use to evaluate changes to that model is four or five criteria like these, not a single accuracy percentage.
Pull adversarial cases, not a random sample
A representative sample tells you how the feature performs on average. It almost never tells you how it fails, because failures cluster in a small number of input shapes: the scanned page that's slightly rotated, the invoice with a currency symbol the parser hasn't seen, the customer name that's also a product SKU. Twenty cases picked because they already broke something, or because you know they're structurally different from the happy path, will surface more real risk than five hundred cases pulled at random.
The twenty-case gate
The mechanism we use is simple enough to run before every model or prompt change, not just before a launch:
- Pull 15–25 real inputs: half from cases that already failed in production or QA, half from known edge categories (empty fields, non-English text, malformed input, boundary values).
- Run every candidate — old prompt, new prompt, old model, new model — against the same set, side by side.
- Score each case against the rubric, not a vague 'better/worse' judgment.
- Block the release if a case that used to pass now fails, even if the average score went up.
That last rule is the one people skip under deadline pressure, and it's the one that actually protects you. Average scores hide regressions in exactly the cases you'd most want to catch.
Use an LLM judge, but audit the judge
Once the rubric exists, an LLM can apply it to a hundred production cases faster than a human can apply it to ten. That's the honest case for LLM-as-judge: it's a way to spend a labelling budget of zero dollars and one afternoon on rough coverage, not a way to get scientific accuracy. The judge model has correlated blind spots with the model it's judging — trained on similar data, prone to similar failure modes — so it will sometimes rate a wrong answer as right for the same reason the original model got it wrong.
Two things keep the judge honest. First, anchor every judge call to the same rubric criteria you wrote by hand, not to an open-ended 'rate this 1–5.' Second, pull a random 10% of judge verdicts for a human to check every time you run the gate, and track how often the human overrides the judge. If that override rate climbs, the judge has drifted and you fix the rubric or the judge model before trusting the next run.
{
"case_id": "inv-00231",
"input_hash": "sha256:...",
"rubric_id": "extraction-v3",
"criteria": {
"value_matches_source": true,
"unit_correct": true,
"no_fabricated_line": true,
"confidence_flagged_correctly": false
},
"judge": "gpt-4o-eval",
"human_override": null,
"verdict": "fail"
}Log every case this way, win or lose. Six months from now this log is closer to a real labelled dataset than anything you could have built in the two weeks you didn't have.
Where this approach runs out
This is a gate, not a measurement. It answers 'did we regress on the failures we know about' — it does not answer 'what is our accuracy across the full space of inputs we'll see in production,' and it should never be quoted as if it does. If the feature touches billing amounts, medical fields, or anything that ends up in front of an auditor, twenty adversarial cases and an LLM judge are a starting point, not a defensible number.
A twenty-case gate proves you didn't regress. It doesn't prove you're right.
At that point you're back to building the labelled dataset properly, with human ground truth, inter-rater checks, and enough volume for a confidence interval — the thing there wasn't time for at launch. Budget for it once the feature is load-bearing, the same way you'd budget for retrieval quality work once a chatbot moves from demo to production.
Put the gate in CI, not in a slide deck
The twenty-case gate is only worth building if it runs automatically on every change, not manually before big launches. Wire it into the same pipeline that runs your other tests: new prompt or model version goes in, rubric scores come out, and a regression blocks the merge the same way a failing unit test would. That's the difference between an eval that exists and an eval anyone actually looks at.
The dataset you didn't have time to build in week one gets built anyway, one production case at a time, as a side effect of running the gate every week after. By the time you do have a slide deck moment — a board update, a compliance review — you're not presenting vibes. You're presenting a log with months of comparative verdicts behind it, which is a much better position to defend than a number someone made up under deadline pressure.
Frequently asked
How many test cases do I need to evaluate an AI feature without a labelled dataset?
Fewer than you'd think if the goal is a ship/no-ship decision rather than an accuracy number. Fifteen to twenty-five cases — half drawn from real failures, half from known edge categories — are usually enough to catch a regression between two prompt or model versions before wider rollout.
Is LLM-as-judge reliable for evaluating AI features?
It's reliable for rough coverage at speed, not for a defensible accuracy figure. The judge model shares blind spots with the model it's grading, so anchor it to the same rubric you'd use for human review and spot-check a sample of its verdicts with a person to catch drift.
What's the difference between an eval and a labelled dataset?
A labelled dataset gives you ground truth for every case, built for statistical measurement. A fast eval is a rubric applied to a small, deliberately adversarial set of cases to answer one specific comparative question, like whether a new prompt regressed on known failure modes.
When do I need a real labelled dataset instead of a quick eval?
Once the feature touches money, health data, or anything that lands in front of an auditor, or once volume is high enough that rare failures matter statistically. At that point a twenty-case gate is a starting point, not proof, and you need human ground truth with enough volume for a confidence interval.