Why “It Feels Good” Is Not a Test
Three things are wrong with the nodding-at-the-screen method, and they compound. The sample is tiny, so a 60%-correct system and a 90%-correct system look identical. The sample is chosen by the builder, who unconsciously types the questions the system was designed for. And nothing is written down, so when you change the prompt next week you have no way of knowing whether you improved it or quietly broke it. You are not measuring the system. You are measuring your own optimism.
The honest answer is boring and has been the same for years: fix a set of real cases with known-good answers, run it on every change, and look at the number. Nobody skips this because they have never heard of it. They skip it because building that set means a senior person has to sit down and decide what the correct answer actually is for fifty messy real cases — unglamorous work that nobody budgeted and nobody enjoys. That is the whole obstacle. The metrics are the easy part. The labels are the project.
The Golden Set Is the Whole Job
A golden set is a fixed collection of real inputs with the answer you would accept, agreed in advance. Start with forty to sixty cases, not five hundred — which surprises people who want to do it properly. The reasoning: you will rewrite a third of your first set once you see what the system actually gets wrong, because your idea of a hard case and the system's are different. Start small, run it, then let every production failure become a new case. After six months the set is large, and every row is there because something genuinely broke.
Take a machine builder putting a service assistant on top of twenty years of manuals, so field technicians stop calling the office from the customer's shop floor. The golden set is fifty real questions technicians actually asked, each with the answer a senior technician signs off as correct.
Sample for your failures, not your traffic
Here is the mistake that survives even in teams doing everything else right. It feels rigorous to build the golden set as a representative sample of production traffic. Do not. Real traffic is mostly easy — the same dozen routine questions your system already answers correctly. Build the set that way and most of it is cases you always get right, so your score parks itself around 88% and refuses to move no matter what you change. Meanwhile the cases that generate the angry phone calls are three rows out of fifty, statistically drowned by the easy ones.
Over-sample the hard cases deliberately. For the machine builder: questions whose answer lives in a table rather than a sentence, questions about the variant that shares a name with a different variant, ambiguous questions, and — critically — questions the manuals genuinely do not answer. If you also want to estimate real-world rates, keep a second, smaller representative slice and run it occasionally. Two sets, two jobs: the hard set drives development, the representative set estimates production. Averaging them together gives you a number that does neither job.
Someone has to write the right answer
The correct answer must come from the person who does the work today — the senior service technician, not the project manager, and not another LLM. Cost it honestly: fifty cases at roughly ten minutes each is a day of your most expensive person's time, plus a second day when the set gets revised. That bill is why golden sets do not get built. It is also the cheapest day in the project, because it is the day that stops you shipping a confident wrong answer to a technician standing next to a stopped machine.
Two details pay for themselves. Record why an answer is right, not just what it is — six months later nobody remembers, and the note is what lets someone else maintain the set. And put the unanswerable questions in, with the expected answer being an honest “I don't know, call the office.” A set containing only questions that have answers carefully avoids measuring the failure mode that actually hurts you.
Metrics That Fit the Task
“Accuracy” is not a metric until you say accuracy of what. The useful move is to pick metrics per task, and to accept more than one number:
Extraction — pulling fields off invoices or order confirmations: per-field exact match, plus precision and recall. Weight the fields by consequence. One wrong IBAN is worse than ten wrong free-text descriptions, and a single averaged score hides exactly that. Classification — a regional insurer routing incoming claims correspondence: use the confusion matrix, not one accuracy figure. Which class gets mistaken for which is the actionable part; “91% accurate” tells you nothing you can fix. Retrieval: recall@k — did the right passage even make it into the context. Open-ended generation — drafting a reply: no clean metric exists, and pretending otherwise is where teams start lying to themselves.
The rule underneath: one number per failure mode you care about, not one number overall. A single aggregate score is a score you cannot act on.
LLM-as-judge: useful for comparison, not for scores
Using a model to grade outputs works, within two hard limits. It works for pairwise comparison: given this source passage, which of these two answers is better? It works for narrow, checkable questions: is every factual claim in this answer supported by the passage — yes or no? That grounding check is genuinely useful and cheap to run.
It does not work for absolute scores. A judge that returns “7.5 out of 10” is generating a plausible-looking number, not measuring anything; run it again and you get 8. It does not work when the judge grades its own family's output, which it favours. And it does not work where the judge has no domain knowledge — no model can tell you whether that torque figure is right for that machine variant. Only your technician can. Two practical habits: swap the order of A and B and run it twice, because position bias is real and a verdict that flips means the judge is guessing; and use a different model family for judging than the one under test.
Evaluating the System, Not the Model
The most expensive habit in this field is reaching for a bigger model when the number is bad. Almost always, the model is not the part that is broken — and one end-to-end score can never tell you that, because it collapses every stage into a single figure.
For the machine builder this is not a hypothetical shape. When a service assistant scores badly, the usual culprit is an answer that sat in a table the parser flattened into noise. The fix is a table-aware parser and better chunking: a few days, no extra inference. The model upgrade would have cost more every month, forever, and moved nothing.
Measure both views, always. Stage metrics tell you where to work; the end-to-end number tells you whether the work mattered, because stages that each look fine can still compose badly. Neither replaces the other — and neither survives contact with production unless someone keeps running it after launch.
Prompts and Models Are Code — Test Them Like Code
A prompt is code with no type system and no compiler. Change one word and the effect ripples through outputs in ways nobody predicts by reading it. So treat it accordingly: prompts live in git, not in a database field someone edits on a Tuesday afternoon; every change runs the golden set; outputs are stored so you can diff last week against this week.
The part teams miss: pin the model snapshot. If your code points at a floating alias like “latest”, your provider can move the model underneath you, and your system's behaviour changes on a morning when you shipped nothing at all. Debugging that from the inside is miserable, because every instinct says look at your own diff — and there isn't one. Use dated snapshot identifiers, and treat a model upgrade as what it is: a deploy. Run the set, read the diffs, then move.
Golden sets also rot. When the product changes, some previously correct answers quietly become wrong, and a set nobody has reviewed for a year is worse than no set — it reports green while measuring the wrong thing. Budget a review whenever the underlying process changes.
Where Tippel Fits
Almost none of this is technology. It is judgement: which cases belong in the set, which metric matches the task, which stage is actually losing you points. That is what the AI Readiness Check does on your data — a timeboxed, fixed-price look that ends with a number you can act on, and an honest read on whether the gap to “good enough” is a week of chunking work or a problem the current generation of models does not solve.
If you have a system that demos well and you want to know what it really scores before you build a process on it, that is the conversation. Get in touch.