Questions to ask an AI consulting partner: a one-page checklist
Turn a sales conversation into a set of verifiable decisions about scope, access, testing and ownership.
Written by the Problem Solver Team with AI assistance.
The short answer
The most useful questions to ask an AI consulting partner connect each promise to evidence: what changes in your workflow, what data and permissions are needed, how failures are tested, who makes consequential decisions, and what your team receives at handover. Record each answer as demonstrated, documented or unanswered, then resolve critical gaps before expanding access.
Key takeaways
- Ask every shortlisted partner about the same bounded workflow.
- Separate demonstrated evidence from documents and unanswered promises.
- Treat data access, consequential actions and recovery as decision gates.
- Test the handover as well as the working demonstration.
- Keep a named owner for every unresolved question.
01What should you decide before the first meeting?
Choose one workflow and define the decision the meeting should support. A partner cannot give a meaningful implementation answer if the brief is simply “bring AI into operations.”
Write down the process owner, current inputs, expected output, systems involved and the consequence of a mistake. Bring a sanitized example of an ordinary case and an awkward exception. Do not send sensitive records just to make a sales demonstration more realistic.
Use our first-90-days guide if the process itself is still undecided. If the main uncertainty is whether the job needs an agent at all, read the AI, RPA and agent comparison before discussing products.
The checklist below is our practical synthesis for an initial evaluation. It is not a certification, procurement standard or substitute for your organization's security and legal review.
02Which questions belong on your one-page checklist?
Ask for an observable deliverable alongside each answer. The aim is to leave the conversation with evidence and next decisions, rather than a longer list of features.
| Ask the partner | Request this evidence | Resolve before proceeding |
|---|---|---|
| What exact part of our workflow will change? | A before-and-after process map, including exceptions. | An agreed output and named business owner. |
| Why does this step need AI? | A comparison with a simpler rule, search or automation. | A reason tied to the task, not the product label. |
| What will the system read, store and change? | A data-flow map and permission list. | The permitted data, destinations and actions. |
| Which decisions stay with our people? | An approval and escalation map. | A responsible person and fallback when unavailable. |
| How will we know it is ready? | A test plan with ordinary, difficult and prohibited cases. | Acceptance criteria and a decision owner. |
| How does it fail safely? | A demonstration of a failed dependency and recovery. | A stop procedure, recovery owner and duplicate protection. |
| Who supports it after launch? | A runbook, monitoring responsibilities and change process. | Who investigates, communicates and authorizes changes. |
| What can our team operate or move later? | A handover inventory and a practical walkthrough. | Access, documentation, dependencies and exit responsibilities. |
For each row, add the partner's answer, an evidence link, the remaining gap, an owner and a due date in your own notes. “We will cover that later” can be acceptable during discovery. It should remain an open item, not become a completed check.
03How do you separate a convincing answer from useful evidence?
Classify the evidence by what you have actually seen, then apply a consequence-based gate. Do not let several attractive answers cancel out one unresolved permission or recovery issue.
Use these three labels:
- Demonstrated: you observed the behavior in a relevant, controlled test, including its limits.
- Documented: you received a specific description, diagram or plan, but have not tested it.
- Unanswered: the answer is missing, generic, contradictory or promised for later.
This evidence ladder is an evaluation aid, not a numerical prediction of partner quality. A documented design may be enough to commission further discovery. Access to production systems needs evidence appropriate to the risk and your organization's approval process.
The NIST AI Risk Management Framework is a voluntary framework for incorporating trustworthiness into AI design, development, use and evaluation. Referencing it can help structure a discussion, but saying “we follow NIST” does not prove that a specific control works.
04What should the demonstration include beyond a successful result?
Ask to see a normal case, an exception and a stopped action. A smooth demonstration of the happy path leaves important operating questions unanswered.
Hypothetical example: your team wants help triaging supplier emails. The partner shows a message becoming a draft task with a source link. That demonstrates a narrow extraction and routing step. It does not establish permission to send replies, alter supplier details or approve a transaction.
Next, supply a sanitized message with a missing reference. Watch whether the system asks for clarification or invents one. Then make a downstream service unavailable and inspect the record left for an operator. Finally, repeat the same input to see whether the workflow creates duplicate work.
The OWASP guidance on excessive agency distinguishes excessive functionality, permissions and autonomy. In this example, ask why a reading-and-triage tool would need a sending capability. The partner should explain the actual boundary rather than offer a general assurance about the model.
None of these tests proves the whole system is safe. They reveal which assumptions need a deeper test before rollout.
05What should you ask about support and handover?
Ask who owns the system when the implementation team is no longer in the meeting. Include your operators in the answer, because they will handle exceptions and changes.
The NCSC guidelines for secure AI system development cover the lifecycle through secure operation and maintenance, including logging, monitoring and updates. That makes post-launch responsibilities part of the evaluation, rather than an optional closing topic.
Ask for a walkthrough in which your team finds a failed run, understands what happened, pauses further actions and follows the agreed recovery procedure. Record anything that still requires the partner's involvement. Dependencies are not automatically a problem, but hidden ones make planning difficult.
Clarify which accounts, configuration, documentation and export formats are included in handover. Ownership and usage rights require explicit agreement; do not assume that access to a dashboard gives you ownership of every underlying component.
06How can you turn the answers into a next step?
End with a bounded decision and a written gap list. You can continue discovery without authorizing production access or a wider rollout.
- Compare like with like. Give each partner the same workflow and exceptions. Done when their answers address the same scope.
- Attach evidence. Put the demonstration or document next to each answer. Done when another reviewer can trace the claim.
- Mark critical gaps. Identify missing decisions about data, actions, testing and recovery. Done when each has an owner.
- Choose the next gate. Continue discovery, run a controlled pilot or pause. Done when the required evidence and decision-maker are written down.
Apply the same checklist to ProblemSolver. For an AI Opportunity Audit focused on your workflow, Book a 30-minute call.
07Frequently asked questions
What is the most important question to ask an AI consulting partner?+
Ask what specific workflow will change and what evidence will show that the change is acceptable. This brings the conversation back to your process instead of the technology. Follow it with questions about permissions, exceptions and the person who makes the rollout decision. No single answer replaces the rest of the evaluation.
Should we provide real company data during a sales demonstration?+
Start with sanitized or synthetic examples that preserve the structure of the task without exposing sensitive records. If real data becomes necessary, agree its purpose, destination, access and handling first through your organization's review process. A more realistic demonstration is not, by itself, a reason to grant broad access.
Is a polished AI demo enough to choose a partner?+
A demo can show that a narrow behavior works under the conditions you observed. It does not establish reliability across your workload, secure permissions or support after launch. Ask to see exceptions, stopped actions and recovery, then distinguish those observations from capabilities that are only documented or still unanswered.
What if the partner cannot answer every question immediately?+
Some answers depend on discovery, so an honest unknown is useful information. Record the gap, who will investigate it and what decision depends on it. Continue only with work that does not rely on the missing answer. Do not silently treat a future promise as evidence for granting broader access.
Sources and further reading
- AI Risk Management Framework. NIST, 2023.
- LLM06:2025 Excessive Agency. OWASP, 2025.
- Guidelines for secure AI system development. NCSC, 2023.