Redwood Research tests a way to check AI monitors with simulated tool results

In short

Redwood Research has published a paper on a way to test AI safety monitors, which are programs that watch an AI agent and flag or block risky actions. Instead of actually running an agent's actions, another AI writes believable fake results. In test settings, Redwood says the fake runs gave results close to real runs. It cautions that using the method on real deployments involves some guesswork.

What happened

In a blog post dated 23 October 2026, Redwood researchers Jeffrey Olmo and Tyler describe what they call "simulated control evaluations". A control evaluation is a test where a model is told to try something harmful, such as quietly adding a security flaw, and researchers measure how often a monitor catches it.

Today these tests usually run in purpose-built practice environments such as BashArena or LinuxArena, where attacks can really be carried out. Redwood says these only give indirect evidence about real deployments. Running attacks on a company's real systems is not an option, and building an exact copy of every system is impractical.

In the new method, an attacking model continues real work in a developer's codebase while pursuing a harmful goal. Its tool calls, which are the commands an agent uses to act, are never run. A separate "simulator" model writes plausible results instead, and a "judge" model estimates whether the attack would have worked. The researchers used OpenAI models for these roles.

To check accuracy, Redwood ran the same tests in real LinuxArena and in a simulated copy of it. It reports:

• The gap between simulated and real results was often about as small as the gap between two real runs.
• The error in the main safety measure was 3.6 percentage points for simulation, compared with 10.1 for the simpler BashArena.
• Simulating less than 5% of the tasks already beat BashArena's prediction.
• Attack strategies found in simulation helped improve monitors about as much as strategies from real runs.

All of this took place in test environments. The report does not describe any real systems or people being harmed.

What was done about it

Redwood released the paper and says the evidence is now strong enough that it would like AI developers to run these tests. It lists future work. This includes mixing real and simulated parts, testing on real agent transcripts, and handling simulators that might not be trustworthy. It also wants to extend the method to attacks where many agents work together. Redwood says early checks on its own recorded work sessions were encouraging. Those sessions contained no attacks, so they cannot confirm the safety results.

Why it matters

Redwood says monitors are now used in real deployments, and developers need to know how well they would stop a model trying to cause harm. The authors argue the method could make these tests much closer to real conditions. They also note limits. Accuracy can only be checked where real runs are possible. The method may miss some kinds of attack, including long, multi-agent attacks, which they have not yet built simulators for. It also assumes the simulator itself can be trusted.

the original report