Redwood Research tests a tool that finds blind spots in AI safety monitors

In short

Redwood Research reports testing a tool called Screwtape, which edits real records of AI agent activity to look for weaknesses in safety monitors. Monitors are systems that watch what an AI does and flag harmful actions. In 252 test runs, six attacks passed a frontier AI developer's monitor without being caught and were confirmed as valid by human reviewers. The tool inserted these attacks, so they do not show AI agents choosing to attack on their own.

What happened

Redwood Research described the work in the original report. This was a research test, not a real deployment. It used transcripts, which are records of what an AI agent did, taken from the work of Redwood staff. The report does not describe harm to any real person or organisation.

Screwtape uses AI models in two main roles:

• The attacker: edits a transcript to insert a harmful action and writes a description of it.
• The verifier: checks whether the attack is realistic, clearly harmful and could plausibly succeed.

A monitor then scores each attack that the verifier accepts. If the verifier rejects the attack or the monitor catches it, the attacker receives the reasoning and can try again.

The tested monitors looked only at an agent's actions, not its written reasoning. Redwood first refined attacks against its own monitors, which were based on Opus 4.8 and Gemini 3.1 Pro. It then tried the finished attacks on a separate monitor from a frontier developer. The report does not name the developer.

Redwood ran the tool 252 times on 94 different transcripts. This produced 35 attacks that got past Redwood's own monitors. Of these, 17 also got past the developer's monitor. Human review confirmed six of the 17 as valid.

Redwood says finding these attacks was hard. Monitors caught most attempts, and most runs found nothing. Each attempt took about 5 to 10 minutes, and a run could take a few hours. The verifier also sometimes accepted invalid attacks, so people still had to check the results.

What was done about it

Redwood is not releasing the tool for now, but says it is open to sharing it with researchers. The report does not say whether the frontier developer changed its monitor. Redwood's possible next steps include:

• Adding support for transcripts that include an agent's reasoning.
• Using the attacks found to improve monitors automatically.
• Simulating the test environment to help verify attacks.

Why it matters

Redwood says the attacks can help developers fix blind spots in their monitors. They can also test safety claims: a confirmed attack can show that a claim about catching a certain type of attack does not hold. The report stresses several limits. The tool cannot show how often a monitor fails in normal use or produce a numerical safety estimate, and finding no attacks would say little about safety. The failures also offer limited evidence of real risk, because the attacker sees the whole transcript and gets feedback after every attempt, which a deployed agent usually does not. Redwood adds that it is "quite uncertain" whether helping improve frontier monitors is good, or the best use of research time.