A Production AI Agent Checklist: Identity, Evaluation, and Recovery

Why the release checklist needs to change

An assistant that drafts a paragraph can be wrong without changing a system. An agent that updates an account has a different failure boundary: its output becomes an input to an API. A useful review therefore asks what the application can do when the model misunderstands a request, receives hostile retrieved text, or loses a connection halfway through a task.

Current platform guidance points toward stronger operational controls. Microsoft distinguishes user-delegated agents from autonomous agents with their own identities. Databricks argues for shared infrastructure for agent context, governance, evaluation, and observability. Those are product directions, not evidence that adopting a platform automatically makes an application reliable. The checklist below is a proposed way to turn the operating concerns into release evidence.

1. Define the smallest useful task

Consider an illustrative invoice assistant. Its first job is to find overdue invoices for the signed-in account manager and draft a reminder. Sending the reminder is a separate action. Changing payment details is outside its scope. This boundary makes success measurable: the correct invoices, for the correct account manager, with accurate amounts and dates, in a draft that can be reviewed.

• Write down the allowed inputs, outputs, and side effects.
• Identify the deterministic calculation or query behind each factual answer.
• Define when the system must ask for clarification, decline a task, or hand it to a person.

2. Put authorization at the tool boundary

The API must derive the caller and permitted accounts from trusted identity context. Do not trust an account ID merely because the model supplied it. Filter records before returning them to the agent, and check authorization again when an action is executed. An earlier successful read does not prove that a later write is still permitted.

For the invoice example, expose a narrow draft-reminder operation rather than arbitrary SQL or a general outbound HTTP tool. If sending is later enabled, approval should bind to the actual recipients, invoice IDs, and message content. A changed draft invalidates the earlier approval. Treat retrieved documents as data; an instruction inside an invoice cannot expand the application’s authority.

3. Evaluate the task, not just the answer

Build a small, versioned test set before changing prompts or models. Record expected records, tool calls, and side effects alongside the expected response. Use deterministic assertions for account boundaries and duplicate sends. Use human review for whether the draft is clear and appropriately qualified. A fluent answer should never compensate for an unauthorized read.

• A valid request returns only invoices visible to the caller.
• A forged account ID is rejected even when the requested invoice exists.
• An invoice containing instructions to send data elsewhere does not trigger another tool.
• A request with no matching invoices produces an explicit empty result rather than an invented balance.
• Access revoked between drafting and sending prevents the send.
• A timeout followed by retry cannot produce two reminders.

4. Design recovery before enabling writes

Assign a stable operation ID when a send is approved. Store its state at the service boundary and reuse the same ID on retries. If the downstream provider cannot deduplicate requests or report whether a send succeeded, an ambiguous timeout should become a reconciliation task rather than an automatic resend. The system needs a way to distinguish “failed” from “outcome unknown.”

Bound runtime, tool-call count, and spending per task. Save enough structured evidence to reconstruct a failure: model and prompt versions, permitted operation, authorization result, approval reference, and tool status. Avoid recording full invoice contents by default. Test a kill switch that disables new writes while preserving the evidence needed to investigate.

5. Make rollout a sequence of gates

Begin with offline fixtures, then a read-only pilot, then drafts reviewed by people. Enable a narrow write operation only after its failure tests pass and an owner accepts the remaining risks. Set thresholds appropriate to that task rather than copying a universal accuracy target. Track task completion and time saved alongside incorrect actions, escalation rate, latency, and cost per successful task.

The release artifact should be a short evidence bundle: the task boundary, permission map, evaluation results, recovery procedure, and named operational owner. A new model or tool permission changes that evidence. Re-run the affected tests before expanding the rollout.