An AI pilot needs agreed tasks, representative evaluation cases, error categories and a decision rule before the demonstration. Define success for the complete workflow: an acceptable result, appropriate handling of uncertainty, permitted actions and tolerable operating effort. A single average score cannot establish readiness if it hides consequential failures or excludes human correction costs.
Let’s plan your AI pilot.
Bring your workflow and the decisions you need to make. We’ll help you define the requirements, examine the trade-offs and plan a practical next step.
Specify the decision the pilot will support
Write whether the exercise will decide feasibility, architecture, supplier selection or limited operational release. Those decisions require different evidence. A prototype that tests extraction accuracy may not test permissions, monitoring or recovery.
Define the task population and the exclusions. Include expected languages, input types, user groups and sources only where the pilot actually covers them. Do not present a narrow test result as proof about every workflow the eventual product might serve.
Build an acceptance worksheet
| Dimension | What to define | Evidence to retain |
|---|---|---|
| Task quality | Required result and unacceptable errors | Inputs, expected outcomes and scored outputs |
| Abstention | When to ask, defer or escalate | Incomplete and conflicting cases |
| Authority | Allowed data and business actions | Access and action-boundary tests |
| Speed | Useful end-to-end response requirement | Timing for representative cases |
| Cost | Cost per accepted completed task | Usage, retries and review effort |
| Recovery | Behaviour when dependencies fail | Interruption and recovery records |
| Operation | Who owns unresolved work | Queue, escalation and support walkthrough |
Agree thresholds with the accountable business and technical owners. The table is a planning tool, not a universal pass standard or certification checklist.
Make scoring reproducible
Write a rubric that another reviewer can apply. For a document task, distinguish a missing field from an incorrect amount, incorrect customer association or unsupported inference. Define partial credit only where it reflects useful work.
Use a development set to improve the system and a held-out set to assess it. Track dataset versions and record disagreements between reviewers. A score against frequently reused examples can overstate performance on unfamiliar work.
Show how averages can hide failures
Imagine a hypothetical test containing 80 ordinary tasks and 20 exception cases. If all ordinary tasks pass and all exceptions fail, the overall pass rate is 80%. That result gives no evidence that the exception path works.
Now suppose exceptions include cross-account data access. A high overall score cannot compensate for that failure. Report category results, consequences and observed counts alongside the aggregate. These invented counts illustrate reporting; they are not a recommended test size or acceptance threshold.
Compare against the operating baseline
Record how the task is completed today, including review, delay and correction effort. A model response produced quickly may still increase total handling time if staff must check every statement against source records.
Test the proposed human review arrangement as part of the workflow. Reviewers need enough context to detect mistakes and a clear route to correct them. Do not treat review as an unlimited resource outside the cost model.
The NIST AI Risk Management Framework provides broader risk-management context. This acceptance worksheet does not establish conformity, certification or legal compliance.
Questions sponsors ask
What accuracy percentage should we require?
Choose thresholds by task and consequence. Define what an error means before selecting a percentage, and retain enough category evidence to interpret the result.
Can a failed pilot still be useful?
Yes, if it resolves the intended decision and records why the approach failed or needs a different boundary.
Before approving the pilot
Bring the decision, task population, rubric and unacceptable outcomes. Pair this with the evaluation-harness guide and AI supplier checklist when requesting a scoped proposal.