AI project risk management: what can go wrong and how a rollout limits it
A practical checklist for identifying, testing and responding to AI project risks before they become operational surprises.
Written by the Problem Solver Team with AI assistance.

The short answer
AI project risk management starts by naming the business decision, the possible failure, the owner and the safe response. A useful rollout does not try to eliminate all uncertainty. It limits scope, tests realistic cases, monitors the live workflow and pauses expansion when evidence no longer supports the intended use.
Key takeaways
- Treat unclear policy and bad handoffs as project risks, not only model errors.
- Give each material risk an owner, evidence signal and safe response.
- Test stale data, conflicts and uncertain writes before expanding scope.
- Measure workflow outcomes by route and case type.
- Define the pause and recovery rule before release.
01What can go wrong in an AI operations project?
An AI project can fail before a model makes a visibly wrong answer. The workflow may use an unclear business rule, read an out-of-date source, send work to an unattended queue, or repeat an action after an uncertain result. Those are operational risks because they change what a team does next.
Start with a bounded decision, not a broad goal such as “make operations smarter.” Write the intended outcome, the authoritative source, the action the system may take, the action it may only propose, and the person accountable when the evidence is incomplete. The first-90-days guide helps choose that boundary.
NIST's AI Risk Management Framework organizes ongoing work around governing, mapping, measuring and managing risks. For an operations team, that is a useful sequence for making uncertainty visible; it is not a certification checklist.
02What should the project risk register contain?
Keep a small risk register that the process owner can actually use. Each row should state the scenario, the operational consequence, the evidence that would reveal it, its owner, and the response. Avoid a long list of generic “AI risk” labels with no connection to the workflow.
| Risk question | Example operational answer | Control to test |
|---|---|---|
| What decision is being supported? | Classify a supplier request for review; do not change the order. | A request never becomes an order update without a separate authorized step. |
| Which source is authoritative? | The order system, not an email summary. | A missing or stale record changes the case to unknown. |
| What could be wrong? | The reference maps to two orders. | Route to reconciliation instead of choosing one. |
| Who owns the response? | Procurement resolves identity; an authorized manager decides commitments. | The queue has a named primary and backup owner. |
| What is the safe state? | Keep the existing commitment unchanged. | A timed-out write is reconciled before retry. |
This is an illustrative workflow design, not a client result. The point is to make the response specific enough that a team can rehearse it.
03Which risks deserve attention first?
Prioritize by consequence and exposure, not by how dramatic the failure sounds. A rare wrong label may be low consequence if a person reviews it before use. A routine missed handoff may matter more if it silently creates a customer commitment.
Ask these questions for every proposed action:
- Could it change a business record, spend money, promise something externally, or affect a person’s access?
- Can a person inspect the evidence before that consequence occurs?
- What happens if the input is missing, stale, conflicting or malicious?
- Can the team tell whether an attempted write actually succeeded?
- Can the workflow be stopped without losing the cases already in progress?
The human-approval guide explains why a model’s ability to propose an action is not permission to execute it. OWASP’s guidance on excessive agency likewise recommends narrowing available functions and retaining controls outside the model.
04How should a team test a risk before go-live?
Test the difficult path deliberately. A polished demonstration normally proves only that the happy path works with prepared inputs.
Build a small evaluation set from representative, approved cases. Include a routine case, an incomplete request, conflicting records, a stale source, a duplicate event, an unauthorized request and an ambiguous execution result. Define what the system should show, who should receive the case and what it must not do.
For every test, capture the input category, evidence used, route selected, any proposed action, reviewer outcome and final state. Do not call a test successful simply because the model produced a plausible explanation. It is successful when the case ends in the safe, intended operational state.
NIST recommends measuring systems before deployment and regularly in operation, including documented metrics, testing and risk tracking. The AI agent evaluation guide provides a manager-friendly way to turn those cases into acceptance criteria.
05What should be monitored after release?
Monitor both the model output and the workflow around it. A system can be accurate on a narrow classification while still creating risk if cases wait too long, reviewers override it without explanation, or an integration fails silently.
Use measures with clear denominators. Examples include the share of eligible cases that reached review, the share that entered an exception path, time to acknowledgment, unresolved cases beyond an agreed window, duplicated actions, and confirmed incorrect actions. Examine by route and case type. A good result on easy cases should not hide a weak result on incomplete or high-consequence cases.
Set an intervention rule before launch: who can pause the workflow, which signal triggers review, and how cases are handled while it is paused. A pause is a control, not evidence that the project failed.
06What should the process owner ask before expanding scope?
Before adding integrations, permissions or new case types, ask:
- Has the current route worked on representative normal and exception cases?
- Can the team explain the source, owner and safe state for every material action?
- Are overrides, delays and errors being reviewed rather than merely counted?
- Does the next permission change add a measurable benefit that justifies its consequence?
- Can the team restore or reconcile work if the downstream system is unavailable?
Start with an internal, reversible action such as preparing a decision brief or opening a review item. Expand only after the team has evidence that the existing boundary works. To map that first release with your own operations team, book a 30-minute AI Opportunity Audit.
07Frequently asked questions
Is AI project risk management only about model accuracy?+
No. Accuracy matters, but an operations rollout also depends on source freshness, permissions, queue ownership, exception handling and whether an action can be reconciled. A model can produce a reasonable answer while the surrounding workflow still creates duplicate work or makes an unauthorized change.
What is a safe first AI action in operations?+
A useful first action is usually internal and reversible: gather approved evidence, extract a proposed field, or prepare a review item. Keep consequential decisions and external commitments with the authorized person until the team has tested the workflow, handoff and recovery path.
When should an AI rollout be paused?+
Define the pause condition before launch. It may be an unresolved source outage, a rise in confirmed incorrect actions, an unowned exception queue or a control that is no longer working as intended. Pause the affected route, preserve cases and reconcile them before expanding or retrying.
Sources and further reading
- AI Risk Management Framework. NIST, 2023.
- AI RMF Core. NIST, 2023.
- Measure. NIST.
- LLM06:2025 Excessive Agency. OWASP, 2025.