How to Choose an AI Development Company: Evidence to Request Before a Pilot
Evaluate an AI development company with a task-based demonstration, evaluation evidence, data-flow review, operating-cost model and ownership checklist.
Choose an AI development company by how it defines, tests and operates your business task. Request a scoped demonstration, an evaluation method, a data-flow explanation, an operating-cost model and a handover plan. A polished chat interface or a long list of supported models does not answer those questions. This guide provides a buyer's evaluation process. It does not rank suppliers or claim that a checklist certifies an AI system as safe or suitable for every use.
Let’s plan your AI project requirements.
Talk through your requirements with Drema. We’ll help you clarify the scope, explore the trade-offs and plan the next steps for your project.
Give every supplier the same task
Describe the process you want to improve, the users involved and the output they need. Explain which records the system may access and whether it can make changes. Name the person who can decide if an outcome is correct.
Prepare a small set of authorised examples for the initial discussion. Include an ordinary case, missing information, conflicting information and a request the system should decline or escalate. These examples are a starting point for scoping, not proof that a small demonstration represents the whole workload.
Ask suppliers to explain what remains untested. A credible answer can include uncertainty, additional data needs or a recommendation to use a simpler workflow without AI.
Request an evidence pack
| Area | Evidence to request | Warning sign to investigate |
|---|---|---|
| Task definition | Inputs, output requirements and completion criteria | “Accuracy” has no agreed meaning |
| Evaluation | Test-set provenance, scoring and failure categories | Only selected successful examples are shown |
| Data flow | Sources, destinations, access boundaries and retention arrangement | Nobody can explain where the data goes |
| Integration | Permitted actions, approvals and recovery path | The model is given broad unexplained authority |
| Operations | Cost assumptions, monitoring and incident ownership | The pilot has no production operating plan |
| Handover | Repositories, configuration, documentation and accounts | The buyer cannot describe how another team would take over |
Record unanswered questions. A security label or a provider partnership should not replace an explanation of the particular system being proposed.
Evaluate the evaluation method
Ask what kinds of failure are counted and who judges them. A support assistant might need separate checks for unsupported claims, incorrect source selection and failure to escalate. A document-processing workflow may need field-level validation and a review process for uncertain results.
Do not compare supplier percentages unless they used equivalent tasks, datasets, scoring rules and operating conditions. “95% accurate” can describe very different things. Ask for the denominator, exclusions and the treatment of incomplete or unusable results.
Request a baseline comparison. The relevant alternative may be the current manual process, a search tool or a deterministic rule. The purpose is to establish whether the proposed system helps the business task under the agreed conditions.
For broader context, NIST's AI Risk Management Framework provides voluntary guidance for incorporating trustworthiness considerations into AI development and evaluation. Referencing that framework is not evidence that a supplier has passed a certification or that a specific deployment meets a legal obligation.
Run a buyer-controlled demonstration
Agree the task and permitted data in advance. Ask the supplier to show the output, the supporting evidence where applicable, and the system's response to a difficult case.
Use this demonstration script:
- Complete an ordinary task and explain the completion criteria.
- Submit incomplete information and show the clarification or escalation path.
- Introduce conflicting records and show how uncertainty is handled.
- Attempt an action outside the user's permission and inspect the boundary.
- Make a dependency unavailable and show the recovery behaviour.
- Inspect the records needed to understand what happened.
Use synthetic or appropriately authorised data for demonstrations. Do not send confidential business records to an unknown supplier simply because the demonstration is described as a sandbox.
Compare commercial scope and operating responsibility
Separate discovery, implementation, evaluation and recurring operations. Identify who pays for third-party services and who can change their configuration. Ask how usage growth, longer inputs, human review and repeated attempts affect the budget.
A pilot proposal should state the question it is meant to answer and the decision that follows. A production proposal also needs deployment, access, monitoring, support and handover responsibilities. Do not assume that a successful pilot includes those services automatically.
Use a decision record with evidence rather than an invented universal score:
| Requirement | Evidence received | Open question | Owner and next action |
|---|---|---|---|
| Task acceptance | Agreed examples and scoring method | Are important cases missing? | Buyer and supplier review the sample |
| Data boundary | Architecture and access description | Which external services retain inputs? | Supplier confirms actual terms |
| Operating budget | Stated usage assumptions | What happens at higher volume? | Buyer tests an alternative scenario |
| Handover | Proposed deliverable list | Can another team operate it? | Agree a handover demonstration |
The rows are a template, not evidence that a particular vendor has supplied those materials.
Choose a pilot with an explicit decision point
Define what would justify continuing, changing the approach or stopping. Agree how incomplete evidence will be handled. The decision should consider task quality, failure consequences, operating burden and costs together.
For example, a pilot may show that retrieval over existing documents is useful but that source ownership is unclear. The next step might be fixing the source process, rather than building a larger conversational interface.
Questions buyers ask
Should we choose the supplier using the most advanced model?
Model choice is one design decision. Evaluate the complete workflow against your requirements, including quality, permissions, latency, cost and operational support.
Do we need an agent rather than a chatbot?
First define whether the system needs to retrieve information, draft an answer or execute actions. Ask suppliers to justify additional autonomy and explain the approval and recovery process. The label alone does not establish a business benefit.
What should we prepare before contacting a supplier?
Bring a task description, authorised sample inputs, the desired output, the current process and the systems involved. Identify the business reviewer and the failures that must be escalated.
Make the first conversation concrete
Use the evidence pack to discuss AI development and the relevant integration requirements. Agree what the first assessment should prove before expanding the build.
