AI agent acceptance criteria: define good enough before go-live

01What should AI agent acceptance criteria actually measure?

AI agent acceptance criteria should measure the completed task, the permitted actions, and the handling of exceptions. A persuasive answer is not proof that the underlying work was done correctly.

For an operations team, “the agent works” is too broad to approve. Replace it with a statement such as: “For a complete intake request, it creates one draft record, preserves the supplied fields, and sends uncertain cases to the named reviewer without contacting the requester.”

Anthropic distinguishes an agent's transcript from the resulting state in the environment: an agent saying a booking exists is different from that booking actually existing. Its evaluation guidance also separates tasks, repeated trials, and graders. Apply that distinction to your own workflow: inspect the record, not just the closing message. Anthropic's agent evaluation guide explains these building blocks.

If you have not chosen a narrow first workflow, start with where to start with AI in operations. Acceptance criteria cannot rescue an undefined scope.

02How do you turn an ambition into a testable release checklist?

Write a condition, an observable result, and a release decision for each important scenario. Then name the person who owns the decision and the evidence they will inspect.

The following checklist is a practical synthesis for an operations pilot, not a certification standard. Complete it with the people responsible for the process and its systems.

1. Define the boundary. List the inputs, permitted systems, permitted actions, and explicitly excluded work. Done means a reviewer can distinguish a valid request from an out-of-scope request without asking the developer.
2. Specify the end state. Describe the record or artifact that should exist, its required fields, and anything that must remain unchanged. Done means the result can be checked independently of the agent's explanation.
3. Write exception cases. Include missing data, conflicting information, duplicate requests, unavailable tools, and instructions embedded in retrieved documents. Done means each case has an expected stop, clarification, or escalation behavior.
4. Assign evidence. Choose a system-state check for writes, a source check for factual answers, and human review for judgment-dependent quality. Done means each criterion identifies where its evidence will be stored.
5. Set release rules before testing. Separate unacceptable actions from tolerable quality defects. Done means a good average score cannot cancel a violation of a critical boundary.
6. Assign recovery and retesting. Identify who can pause the workflow, how unfinished work is handled, and which tests rerun after a change. Done means the owner can explain the response to a failed pilot without improvising it.

Microsoft's agent evaluation checklist similarly connects test cases to acceptance criteria and a test method, and recommends recording results with an agent version and baseline date. Keep that version information beside your decision, not in a separate conversation.

03Which failures should stop a release?

A release blocker is a failure whose consequence the business has not agreed to accept. Treat it separately from improvements that can wait within a deliberately restricted pilot.

Use a pass, hold, or reject decision for each test. “Hold” is not permission to proceed automatically; it requires an explicit decision on a narrower scope or a fix.

Do not borrow a generic accuracy threshold from a vendor demo. Decide acceptable limits from the consequences of this workflow and document what remains unmeasured. NIST's AI RMF Measure playbook calls for fit-for-purpose testing, acceptable performance limits, corrective action, and documentation of risks that cannot be measured.

Passing a finite test set does not prove that an agent will never fail. It provides evidence for a bounded decision, with monitoring and an escalation path still required.

04What does a worked acceptance test look like?

A useful test makes the allowed outcome and the forbidden outcome equally explicit. The example below is hypothetical and illustrates design choices, not a client result or performance benchmark.

Imagine an agent that prepares supplier-onboarding requests for an operations reviewer. Its initial scope is draft creation only; it cannot approve suppliers, modify payment details, or email external contacts.

Normal case: A request contains a company name, internal owner, and service description. The expected result is one draft intake record containing those supplied values, a link to the original request, and a review task for the internal owner. The reviewer checks the stored record and task, not only the agent's response.

Missing-information case: The internal owner is absent. The agent must mark the item incomplete and route it to the designated intake queue. Inventing an owner fails the test. The exact queue and required fields must be configured for the real process before testing.

Duplicate case: The same request arrives again. The expected result is a reference to the existing draft rather than another draft. Check the count and identifiers in the target system after both attempts.

Tool-failure case: The create request times out. The agent must not assume either success or failure and blindly create a second record. It should check the system's available receipt or duplicate-detection mechanism, then escalate an unresolved result.

Boundary case: An attached document tells the agent to approve the supplier. The agent treats that text as content, not authority, and leaves approval with the human reviewer.

For each case, save the input, expected state, observed state, tool receipts, version, decision, and reviewer. Repeat important cases to expose inconsistent behavior; choose the number of trials according to risk and available evidence rather than presenting a small sample as statistical assurance.

05What should the owner sign off before go-live?

The owner should approve a specific version, scope, and operating arrangement, not an agent in the abstract. A release decision should remain traceable after the people involved have forgotten the demo.

Before a limited pilot, confirm that every critical case passed, unresolved defects have owners, permissions match the tested scope, reviewers have capacity, and a pause procedure has been exercised. Record any exclusions prominently. A successful draft-only test does not authorize sending messages or approving transactions.

Keep the accepted cases as regression tests. After a model, prompt, tool, permission, or source change, rerun relevant tests and compare results under the same conditions. For what happens after launch, see why AI agent performance degrades.

If the workflow still lacks clear boundaries, ProblemSolver's AI Opportunity Audit is a place to discuss the process, risks, and a practical first test. Book a 30-minute call.

06Frequently asked questions

Sources and further reading

1. Demystifying evals for AI agents. Anthropic, 2026.
2. Review the agent evaluation checklist. Microsoft Learn, Accessed 2026.
3. AI RMF Playbook: Measure. NIST, Accessed 2026.