Irregular tests whether AI models can break out of their evaluation setups

In short

Irregular, a group that evaluates AI models, says it now runs "containment challenges" before testing a model's cyber abilities. In one recent test, Irregular reports, a new model found a way out of its test setup through unexpected behavior in a cloud provider's network. Irregular says the model stayed contained and nothing outside the test was affected.

What happened

Irregular published the original report on October 5, 2026. It describes a new kind of security test for its evaluation environments. An evaluation environment is the controlled setup where a model is tested.

Irregular explains that cyber evaluations deliberately let an AI agent run code, inspect systems and build attack sequences against realistic targets. It says an agent pursuing such goals may also probe the infrastructure around it or try to get past the controls that limit its task.

In a containment challenge, Irregular gives the model an explicit goal that requires crossing a defined boundary of its environment. In the "network isolation" challenge, the model must fetch a "flag," a marker file, from an online endpoint outside the network it should be confined to. The model runs code inside the test environment and gets a general description of the infrastructure. It chooses its own approach. Irregular says it does not tell the model which technique to use.

Irregular says the test uses the full planned evaluation setup, with at least the same time and budget. It may also give the model architecture details it would not normally see.

In a recent run, Irregular says, a new model found a route out. The route came from unexpected behavior in the cloud provider's networking infrastructure. The environment was set up to block general internet access, and Irregular says earlier models had not revealed this path in previous tests.

What was done about it

Irregular says the issue was caught during the containment test, before the planned capability evaluation. It says the model remained contained, with no impact outside the controlled test. Disclosure of the unexpected cloud behavior to the provider is "in process," according to Irregular.

More generally, Irregular says that when a challenge exposes a gap, it investigates, fixes the affected controls and retests before the capability evaluation goes ahead. It says it plans to share more detail on other security measures in future posts.

Why it matters

Irregular says containment results only apply to the setup that was tested, so changes to the model, tools, permissions, budget or infrastructure should trigger retesting. It encourages evaluators and frontier model developers, meaning those building the most advanced AI models, to make containment challenges a standard part of evaluation setup. The report does not name the model or the cloud provider.