Build an AI evaluation dataset from authorised examples of the tasks the system must perform, with expected outcomes and a documented scoring rubric. Include routine work, meaningful exceptions and cases that should trigger clarification or refusal. Keep development examples separate from held-out evaluation cases so repeated tuning does not become evidence of general performance.
Let’s plan your AI evaluation dataset.
Bring your requirements and the decisions you need to make. We’ll help you clarify the scope, explore the trade-offs and plan the next step.
Define the population before collecting examples
List the intended users, document types, languages, sources and operating conditions. Record what the first release excludes. A dataset of clean short requests cannot establish performance on long, incomplete submissions merely because both are labelled support questions.
Collect representative work and known failure cases for different purposes. Representative sampling helps describe the intended population; a deliberately difficult regression collection helps detect recurring failures. Report those collections separately when their sampling methods differ.
Create a reusable case record
| Field | What to record | Why it matters |
|---|---|---|
| Case identifier | Stable reference and version | Trace a changed result |
| Input | Authorised request and relevant context | Reproduce the task |
| Expected outcome | Required facts, action or abstention | Define acceptance |
| Category | Task type and exception class | Explain aggregate results |
| Evidence | Source supporting the expected result | Resolve reviewer disagreement |
| Consequence | Effect of the relevant error | Prioritise failures |
| Review status | Reviewer decision and uncertainty | Avoid false certainty in labels |
This is an original case template, not a fixed-size dataset prescription. The number of cases should reflect task diversity and the decisions the evaluation must support.
Write labels that a reviewer can apply
For extraction, identify the field, type and accepted representation. For generated explanations, specify supported facts and unacceptable unsupported claims. For action-taking systems, separate correct recommendation, permitted action and successful execution.
Allow an unresolved label where reviewers cannot agree on the business rule. Do not force a definitive answer merely to increase dataset size. Resolve the underlying requirement or keep the ambiguity visible when interpreting results.
Prevent accidental leakage
Group related documents, near duplicates and repeated conversations before splitting development and evaluation data. Otherwise the system may encounter almost the same case during improvement and assessment.
Keep expected answers out of the request context unless supplying them is genuinely part of the task. Review examples generated from templates for superficial patterns that reveal the answer without testing the intended capability.
Access controls apply to the evaluation store as well as production. Use authorised data, minimise unnecessary personal information and document who can inspect case content and results.
Maintain a versioned review queue
When a production failure arrives, record the original context and investigate its cause before adding a case. Some failures require a changed requirement, a corrected label or a new exception category rather than another copy of an existing example.
Store evaluation results with dataset, prompt, model and relevant configuration versions. If a judge model scores responses, compare its decisions with human review on representative cases before relying on its score. There is no universal agreement percentage that makes every judge suitable.
Questions teams ask
Can synthetic examples replace real tasks?
They can explore controlled scenarios, but they do not establish how the system performs on the real incoming population. Identify their provenance and purpose.
Should every failure become a test?
Triage it first. Preserve useful coverage without allowing repeated easy variants to dominate the reported score.
Prepare the first dataset
Bring authorised task samples, expected outcomes and unresolved business rules. Use the evaluation-harness guide for execution tooling and the AI acceptance worksheet for release decisions.