In short
- A 90% accuracy measured on 50 examples could really be anywhere from about 79% to 96%; on 300 examples the range narrows to about 86% to 93%, which is precise enough to decide whether to ship.
- The examples should be real inputs drawn at random from a recent period, with rare but costly cases added as a separate slice and reported separately.
- Deciding the correct answer for each example forces decisions the business has never written down, so the labelled set becomes the specification for the system.
- The golden set should never be used to tune prompts, only to measure them, or the accuracy it reports stops meaning anything.
The first deliverable of an AI build should be a golden dataset, not a model: a few hundred real examples, each with the answer the business agrees is correct. We made that argument in why most AI pilots never reach production. This article is about the number, and about the part most advice on evaluation skips. The examples are easy to collect. The answers are hard to write.
A running example helps. Take a property management firm that wants a model to sort the emails arriving in its shared inbox from tenants and leaseholders into urgent repairs, routine repairs, rent or service charge queries, complaints and everything else. The problem has the same shape for a wholesaler reading emailed orders or a practice sorting the documents clients send in.
Why 300, and not 30 or 3,000
The dataset exists to produce a number, and a number from a small sample is mostly noise. If the model gets 45 of 50 examples right, the measured accuracy is 90%, but the Wilson score interval puts the true figure anywhere between about 79% and 96% at 95% confidence. That range contains a system you would ship and one you would not.
At 300 examples the same 90% narrows to roughly 86% to 93%. At 1,000 it is 88% to 92%. Precision improves with the square root of the sample size, so the first few hundred examples buy most of it. Going from 300 to 1,000 costs more than three times the labelling and only halves the range.
The overall number also hides the categories. Spread evenly across five categories, 300 examples gives 60 each, and a 90% measured on 60 examples could really be anywhere from 80% to 95%. If one category matters more than the rest, it needs more examples than an even split would give it.
Real examples, sampled on purpose
The examples have to be real inputs from the business’s own inboxes, forms or files. Examples written for the purpose, by a person or generated by a model, are cleaner than the real thing: fewer typos, one request per email, no forwarded chains, no photograph of a handwritten note. A system tested on tidy examples gets a tidy score. Anthropic’s guidance on building evaluations puts the rule plainly: design them to “mirror your real-world task distribution.”
Real does not mean the examples someone remembers. People remember the memorable cases, which are rarely typical. Draw the examples at random from a recent period, the last three months say, so the mix of categories matches what the system will actually see.
Then add the rare cases deliberately. In the letting inbox, a report of a gas smell or of water coming through a ceiling may be one email in several hundred. A random sample of 300 might contain none, and the system’s handling of the most important emails would go unmeasured. Add them on top as a separate slice and report the two numbers apart: accuracy on the random sample tells you how the system does on an ordinary day, and accuracy on the rare slice tells you whether it can be trusted with the emails that matter. Blending them produces a figure that describes neither.
Writing the answers is writing the specification
Collecting 300 emails takes an afternoon. Deciding the correct answer for each is where the project really starts, because it forces decisions the business has never had to write down. Is a tenant chasing the same repair for the third time a repair or a complaint? Is a leaseholder disputing a service charge a query or a complaint? The team has been handling these by judgement for years, and the judgement is not written anywhere.
This is a documented effect. A 2024 study of people grading model output, Who Validates the Validators?, found that “users need criteria to grade outputs, but grading outputs helps users define criteria.” The criteria moved as people saw more examples. Expect the first 50 answers to be revised by the time you reach 300, and plan a second pass.
A golden dataset is not a test of the specification. It is the specification, written down one example at a time.
Two rules follow. First, one person has the final say. Hamel Husain’s guide to evaluating LLM products argues for a single principal domain expert, because different people in an organisation judge the same output differently. Second, before that person decides, ask someone else to label a sample of 30 to 50 independently. Where the two disagree, the fault is not with the labeller. The process itself is ambiguous, and a model will be exactly as inconsistent as the people are. Those disagreements are often the most useful output of the exercise, and some are worth settling before any model is built.
Keep each answer as simple as the task allows: the right category, the right product code, a pass or a fail. A score out of five invites argument about the difference between a 3 and a 4, and it cannot be checked automatically.
What it costs
The cost is expert time, not engineering time. If the correct answer for a typical example takes two minutes to decide and record, 300 examples is ten hours, before the second pass and the conversations about disagreements. Call it two to three days of the person who knows the work best, spread over a fortnight. That is the expensive part, because it is usually the person the business can least spare.
It is still cheap next to the alternative. Without the dataset, the only way to know whether the system works is to put it live and watch, and every later change, whether a new model, a shorter prompt or a cheaper provider, gets tested on customers. Anthropic’s guidance favours more test cases graded automatically over fewer graded by hand. That is right, and it only works if there is a set of known-correct answers for the automatic grading to compare against.
Keeping it honest
A golden dataset stays useful only if it is protected from the build.
- Do not tune on it. If prompts are adjusted until the 300 pass, the 300 stop measuring anything. Keep a separate working set for iteration and run the golden set only to measure.
- Version it. When an answer changes because the business changed its mind, accuracy before and after the change is no longer comparable, and the record should say so.
- Grow it from production. Every case the live system gets wrong, once corrected, is a candidate for the next version. After six months the dataset should describe the work better than any procedure document does.
- Treat it as live data. Real tenant emails contain names, addresses and sometimes health details. The dataset is personal data under UK GDPR and needs the same care as the inbox it came from, which we covered in where your data goes when you use a language model.
None of this is sophisticated. It is the discipline that lets a business answer the one question about an AI system that matters, how often it is right, with a number rather than an impression. It is how we intend to start every AI automation build, and for a property firm or anyone else with an inbox full of judgement calls, the labelling is worth doing even if the model is never built, because it writes down how the work is actually done.
Common questions
How many examples do you need to evaluate an AI system?
Around 300 real, labelled examples is a practical starting point. At that size a measured 90% accuracy is precise to roughly three or four points either way, while 50 examples leaves a range of about 79% to 96%.
Can I use synthetic or AI-generated examples for an evaluation dataset?
Not for the golden set. Written or generated examples are tidier than real inputs, so they produce a flattering score. Draw real inputs at random from recent work and add rare cases deliberately.
Who should label an AI evaluation dataset?
The person who knows the work best should have the final say on every answer. A second person labelling a sample of 30 to 50 independently shows where the process itself is ambiguous.
How long does it take to build a golden dataset?
At about two minutes an example, 300 examples is roughly ten hours of expert time, plus a second pass and time to settle disagreements: two to three days of the domain expert’s time, spread over a fortnight.
Written by the Voxalone team. If this raised a question about your own systems, get in touch — you will get a straight answer from an engineer, with no obligation attached.