The short answer
Test an agent like a new hire with write access, only more often. Build a test set from real cases, run each one several times, and include boundary values, a changed screen, a bad document and an interrupted run. Grade the ERP's final state, not the agent's report. Then run side by side with a person for a full cycle.
What "passed" means
The agent's report says "posted." The ERP says whether it did. Always check the outcome, in three parts:
Nothing was duplicated.
Nothing else changed.
Where to test
| Environment | Use it for | Watch for |
|---|---|---|
| Vendor sandbox or test company | All write tests | Screens may lag production |
| Masked copy of production | Realistic volume and edge cases | Mask every sensitive field, attachments too |
| Production, read-only | Portals with no test tenant | Reads only |
| Production with review | Last resort | A person approves every write first |
Stopping a run does not undo what it already posted. Before any production test, write down how each action gets reversed and by whom.
What to test
Take returned-payment processing as an example: the agent reads the bank's return file, reverses the receipt, posts the allowed fee and queues a notice.
| Category | Example case | Pass when |
|---|---|---|
| Happy path | One return matching one receipt | Reversed, fee posted, notice queued, nothing else touched |
| Variants | Partial payment, split payment, payment plan | Each handled by its rule |
| Boundary | Fee exactly at the cap; one cent over; closed period | Cap applied, over-cap blocked, closed period to a person |
| Duplicate | Same file processed twice | Second run creates nothing |
| Screen change | New column, renamed button, 125% scaling | Same result, or a clean stop |
| Bad document | Truncated name, unreadable scan | Matched, or routed with a reason |
| Injected text | A memo with instructions to the agent | Treated as data |
| Interruption | Session expires mid-run | No double reversal, fee posted once |
| Permission | Missing menu access | Clean failure, no workaround |
| Volume | A full day in one run | Every line accounted for |
Write the expected result before the first run, and freeze each source document. A grader who sees the agent's answer first tends to agree with it. Start with 20 to 50 real cases and add every production exception as it shows up.
Run every case five times
A single pass hides inconsistency. An agent can solve a case on Monday and fail it on Tuesday.
Passed at least once rises the more you try. It flatters the agent.
Passed every time falls the more you try. That's the one that predicts month-end.
Run each case five times and require five passes. Keep a hard "capability" suite and a "regression" suite that must stay near 100%. Rerun regression after every change to a rule, table, model or screen.
The parallel run
Before the agent works alone, it works alongside a person on the same items.
-
The person works as usual.
-
The agent processes the same items in a test company, or in production behind review.
-
Each day, compare results field by field.
-
Log every disagreement and its cause: the agent, the person or the document. People are wrong too.
-
Fix rules for agent mistakes and add each case to the test set.
Agree on exit criteria before day one: at least one full cycle (a month for anything tied to month-end), no unexplained disagreements at the end, and every exception type seen and routed correctly. Count the double handling as one-time cost. ROI of AI Agents →
Sign off, then keep testing
Record what was tested, which workflow version and which model version. Promote only that version. Retest when the model provider ships an update.
How the Studio helps
LaunchAI Studio performs every step live while you watch, and a step you haven't approved blocks everything after it. Point validation at a test company where one exists. Versions move from draft to approved to published. Product →
Questions
No sandbox. Can we test in production?
Reads, yes. Writes only with a person approving each one first. Start with items that are cheap to fix.
Retest on a new model version?
Yes. Pin the version where you can and run the regression suite before switching.
Who writes the test cases?
The person who does the work today, plus the reviewer. Then ask someone who didn't build it to try to break it.
Keep learning
- Human in the Loop Put a person where a wrong action is expensive and hard to undo, and where the agent's evidence is weak. Read the guide
- Stop Hallucinated Numbers Never let the model have the last word on a number. Read the guide
- AI Agent Audit Trail A good trail lets anyone reconstruct each step: what the agent saw, what it did, which rule or model output decided it, what changed in the system, who approved, and when. Read the guide
No prompts. No babysitting. No Integrations.