Study finds AI agent test programs can reward runs that make unrequested changes

In short

Richard Abrich of OpenAdapt.AI reports that the scoring programs in two AI agent test suites, WorkArena and AppWorld, passed some test runs that included changes nobody had asked for. The runs were edited by hand as part of a test of the scoring programs. In AppWorld, one added check caught all six duplicate actions without failing any correct runs.

What happened

The paper looks at reinforcement learning, a way of training AI through rewards. In the setup it studies, a benchmark (a standard set of test tasks) includes a verifier, a program that checks the work and supplies the reward.

Abrich says checks for reward hacking, meaning ways of earning a reward without properly earning it, mostly look for runs that pass without doing the task. He asks a different question: what happens when a run does the task but also causes a side effect? A verifier that checks only the requested change might still pass it.

To test this, he used mutation testing, which means making deliberate changes and seeing whether a check notices. He took correct runs and edited them by hand. Some edits left the work missing or wrong, while others added an unrequested change.

A pass counted as a false positive, or wrongly given pass, only if his own check found that the end result was wrong. He reports that every false positive came from an unrequested change. None came from missing or wrong work.

The paper gives these results:

• WorkArena, checked once after submission: The form verifier passed forms both with and without an unrequested field in all 36 valid pairs.
• WorkArena, checked after every step: The verifier passed 7 of 11 runs that filled in the unrequested field last. All of these were text or reference fields.
• AppWorld, repeated actions: Abrich repeated the last write, or saved change, in each run. This changed nothing in 33 of 48 tasks. But in every task from two templates, which are task patterns, the tests passed the duplicate write.

The paper adds that three recent reinforcement learning methods train on one of these benchmarks. It says their reward formulas give the duplicate run full reward.

The work took place in a test setting, using hand-written edits. The paper's summary does not report that any real person or organisation was affected. It also does not describe an AI model making these changes on its own.

What was done about it

Abrich tried a fix in AppWorld by adding one count assertion, a check on how many times something appears, for each template. He reports that this rejected all six duplicates and passed all six correct runs. The paper has been accepted as a poster at the NeurIPS 2026 workshop "Who Verifies the Agents? Toward Reliable Agent Development". The summary does not say whether the benchmarks' makers have changed their tests.

Why it matters

The paper's summary points to a gap in how these rewards are checked. Audits mostly look for runs that skip the task, but a run can complete the task, cause a side effect and still pass. Because the verifier supplies the reward, the paper reports that a duplicate run can receive full reward under the three methods it names. More detail is in the original report.