How to Verify AI Agent Output: 6 Checks Against False Success

An agent's “done” tells you what the agent believes, not what actually happened. So to verify AI agent output, you need evidence from outside the agent. You asked it to reorganize a folder, fix a query or clean up a module. It answered within seconds, the run exited cleanly, and you moved on. Later you found the folder untouched, the number slightly off, or half the code quietly missing.

Each of those failures has a matching check. Here are six of them, each tied to a specific way agents report success that didn't happen.

Some of them take a minute by hand. Others are worth building into your tools once and leaving there.

False success: why an agent's “done” is not evidence

Traditional bugs tend to fail loudly: an exception, a red test, a broken build. Agent failures usually fail plausibly. The output has the right shape, nothing throws, and the summary sounds confident.

A simple example: an agent was asked to file eight documents into folders according to their type. Twenty seconds later it reported success. Not a single file had moved.

Counting errors in tool output show the same pattern in a quieter form. In measured runs where a model had to count matching ids in a tool response, the wrong answers were never wild guesses – they missed by one or two. None of those runs threw an error, and every process exited normally.

In other words, a number that is off by one looks exactly like a correct number.

Edits fail plausibly too. A validator that only asks “is this output well-formed?” never asks “is this still the thing the user had?” An edit can pass every structural check while throwing away most of what existed before.

Check 1: Inspect the end state, not the summary

The first rule: look at the thing the task was meant to change. If the job was to move files, list the folder. A table update means you query the table, and a new API record means you fetch it and see that it's really there. Read the agent's summary last, if at all.

This works well when a task has a clear end state. It gets harder in two cases.

Open-ended requests like “clean up this module” have no single target to compare against. And a state check confirms what you looked for, not what else changed along the way – the file moved, but something else may have been deleted in the process.

There's also a cost argument. Hand-coding a dedicated check per task may take more time than just doing the work yourself. That undercuts the point.

So reserve hand-written checks for tasks where a silent failure would hurt, and reuse generic checks (file listings, row counts, record lookups) everywhere else.

Checks 2 and 3: Compute numbers in code and confirm they reach the model

In one set of runs, a tool returned a list of ids and the model was asked how many matched a condition. In one case it answered 321 while the response in front of it held 322 matching ids. When the model had to count a long list itself, three different agent frameworks all landed at 92.3% accuracy (October 2026).

Here's the key detail: in every wrong run, the tool had been called and its data was sitting in the prompt. The model didn't fail to fetch the data. It misread data it already had.

Check 2: let the tool do the arithmetic. If a question needs a count, a sum or a min/max, compute it inside the tool and return the finished number. Don't hand the model a list and ask it to tally, because that's exactly where the off-by-one answers come from.

Check 3: confirm the number actually arrives. Returning the count from the tool isn't enough if the framework's plumbing drops it on its way from the tool to the model's last prompt. Assert that the computed value is present in what the model finally sees. Then compare the agent's answer against the tool's raw response before you report it to anyone.

Check 4: Catch destructive edits by measuring what the code kept

Edits need a different check, because a destructive edit can be perfectly valid. In one AI app-builder incident (September 2026), a single edit passed validation and shrank 68,000 characters of live JavaScript to 1,400.

After that team switched models, the share of edits followed immediately by a user restoring an earlier version rose from 0.45% (19 of 4,207) to 3.81% (60 of 1,576).

A plain diff (the line-by-line comparison of before and after) doesn't help here: refactoring creates an enormous diff while losing nothing. What works is comparing what survived. Here's the team's rule:

• Extract the declared names from the original – function names in code, element ids in markup.
• Flag the edit as destructive when it keeps under 30% of those names and under 40% of the original size.
• Skip very small files, where ratios mean little.

Both conditions matter. Renaming every function keeps the size. Trimming dead code keeps the names. Only mass deletion drops both.

The same team found a second trap in retries. When an output failed validation, the retry started from that failed output, so the model never saw the original code again.

If a rejection means more than a missing detail, restart from the last good version, not from the output you just rejected.

Check 5: Write the acceptance check before the task, and keep the model out of it

Who writes the check matters as much as what it checks. A test the agent writes after finishing the work tends to share the agent's blind spots: if it misunderstood the task, its test will misunderstand it the same way.

A check written before the task is more independent. But it costs your time on every task.

A related temptation is to let the model decide when the guard doesn't apply – for example, a flag the model sets when it believes the user asked for a full rewrite. That hands the power to grant exceptions to the very part you're trying to guard.

If the model leans toward “this was intentional,” the guard gets bypassed with a label saying everything is fine. A deterministic check – plain code that gives the same answer every time – stops being deterministic once it asks the model for permission.

The workable split: you define the acceptance criteria and the guard up front, as plain code with no model in the loop. The agent can help write that code. But it shouldn't be the one deciding afterward whether its own output passes. For genuine exceptions, rely on an explicit human action rather than a model-set flag.

Check 6: Gate irreversible actions behind new evidence

The last check covers what happens when someone tells the agent its correct work is broken. In a study of this behavior, agents finished a task, passed every check, and were then falsely told by a “teammate” that the results were wrong.

Even the best-performing model wrecked correct work in about one in eight runs. When the false accusation was planted in project instructions or records, models accepted it and made a destructive edit about 40% of the time.

But the urge to “fix” isn't only about accusations. Researchers at ETH Zurich gave agents bugs that had already been fixed, where the right move was to change nothing. Even the strongest agents still made changes in roughly one run out of three.

Two interventions actually helped. In ETH Zurich's tests, asking the agent to reproduce the bug first barely helped. What worked was explicitly allowing the agent to make no change when the issue is already fixed.

The accusation study's own fix was a gate: before the agent could roll back or delete verified work, it needed new evidence or sign-off from a person. With that gate in place, damage dropped by roughly three quarters.

So when you report a problem to an agent, describe what you observe instead of telling it that it broke something. And explicitly allow it to make no change if the issue can't be confirmed.

Key terms

• False success – An agent reporting a task as completed when the intended change did not happen or happened incorrectly.
• End-state check – Verification that inspects what the task was supposed to change, such as a folder, table or record, instead of the agent's report.
• Deterministic guard – A check implemented in plain code with no model in the loop, so its verdict does not depend on model judgment.
• Gaslight sycophancy – An agent's tendency to accept a false claim that it broke something and then damage correct work to “fix” it.
• Restore rate – The share of agent outputs that users immediately undo by restoring an earlier version, used as a signal of bad output.

Calibrate with signals you already have: the 6-check list to verify agent output

Thresholds picked by feel are guesses. If you run agents for other people, your users are already labeling bad outputs for you: every restore, undo, regenerate or abandoned result marks something that didn't work.

Pull those events alongside your check results and tune thresholds against them. That's how the restore-rate jump above became visible – the data had been sitting in the team's own tables.

Here are the six checks, in the order you'd apply them:

• End state: inspect the folder, table or record the task should have changed, and look for unexpected side effects.

• Counts in code: any number the agent reports is computed by a tool, not tallied by the model.

• Hand-off: the computed value is asserted in the final prompt and the answer is compared with the tool's raw output.

• What survived: edits are checked for retained names and size, and retries restart from the last good version.

• Independent acceptance: the check is written before the task, as code, with no model-granted exceptions.

• Gated reversal: deleting or rolling back verified work requires new evidence or human approval, and the agent is explicitly allowed to change nothing.

Check yourself

1. An agent reports it reorganized a folder and lists the new paths. What is the most reliable way to confirm?

• A. Ask the agent to double-check and confirm again.
• B. Inspect the folder itself and compare it to the expected end state.
• C. Check that the agent's response has no errors.

2. Your agent calls a tool that returns 330 ids and then reports how many match a filter. What change removes the off-by-one risk?

• A. Have the tool compute the count and return it, and assert that number reaches the final prompt.
• B. Add “count carefully” to the system prompt.
• C. Switch to a different agent framework.

3. An edit passes all validation but the app's JavaScript went from tens of thousands of characters to about a thousand. Which check catches this without blocking normal refactors?

• A. Reject any edit that makes the code shorter.
• B. A text diff with a maximum size.
• C. Flag only when both most declared names and most of the size disappeared.

4. A user writes “start over and build a simple form.” Why not let the model set a flag that skips the destruction check?

• A. Because flags slow down generation.
• B. Because users never want rewrites.
• C. Because the exception would come from the very component the check guards against.

5. A teammate tells the agent its finished, tested work “breaks the totals.” What setup best limits damage?

• A. Explicitly allow it to make no change, and require new evidence or human approval before deleting or rolling back checked work.
• B. Trust the agent to judge whether the claim is true.
• C. Put “do not break working code” in agents.md.