Apollo Research tests Anthropic's auto mode safety checker for coding agents

In short

Apollo Research, an AI safety organisation, says it ran a pilot test of Anthropic's "auto mode", a checker that decides whether a coding agent's next step should be allowed or blocked. Apollo says it found areas for improvement and suggested fixes, which it says Anthropic put in place. The report does not describe any AI model misbehaving or any real person being affected.

What happened

Apollo Research published the report on 13 July 2026. Apollo says it has spent several years testing advanced AI models before they are released. It is now applying that experience to "monitors". A monitor is a separate system that watches an AI agent and can step in. A coding agent is an AI that writes and runs computer code with some independence.

The system tested was Anthropic's auto mode. Apollo describes it as a new permission mode that decides whether an agent's next action is allowed or blocked. Apollo tested it as a "black box", meaning from the outside, without seeing how it works inside.

Apollo calls the work "red-teaming". That means deliberately trying to find weak spots, the way an attacker might. Apollo says it used three approaches:

• Adding planted problems of different seriousness into records of real coding sessions.
• Running existing fixed test sets, such as one called MonitoringBench.
• A more interactive kind of testing that Apollo calls "dynamic off-policy control red-teaming". The report does not explain this in detail.

Apollo says the purpose of the pilot was to develop its testing methods. The report does not list the specific weaknesses it found.

Apollo says the exercise raised four questions for anyone building monitors:

• Timing. Should a monitor step in when a risky change is first made, or only when it is run or published?
• User permission. What counts as a user truly approving an action, especially during tasks that run for hours or days?
• Boundaries. Where is it safe to send information? Apollo mentions telling public from private destinations, and test systems from live ones.
• What the monitor sees. More detail helps it decide. But Apollo says too much can raise cost and delay, and increase the risk of "prompt injection", which is hidden instructions slipped into the text an AI reads.

What was done about it

Apollo says it paired each area for improvement with a practical recommendation, and that Anthropic implemented them. The report does not say what the changes were. Apollo says it plans to run similar tests on other monitors, starting with major AI companies. A later Apollo post, linked from the same page, says it ran a similar pilot with OpenAI in September 2026. That post says Apollo is now moving its focus to "embedded evaluations", which are tests run from inside the monitoring system.

Why it matters

Apollo says outside testing of monitors can, over time, give stronger safety assurances. It also says such testing can help companies and safety researchers share methods. Apollo notes that monitors are updated on a different schedule from the AI models themselves, so it expects to test them periodically. You can read the original report.