Irregular describes a new test of AI models on multi-step cyber attack scenarios

In short

Irregular, an AI security evaluation group, has described a new test called CyScenarioBench. It measures how well AI models can plan and carry out multi-step cyber attacks in simulated settings. Irregular says the aim is to assess risk. The post is a working draft, and no real organisation was affected.

What happened

Irregular published the post on December 5, 2025. It says it is a working draft that may be revised.

The group says older cybersecurity tests are becoming "saturated" for leading AI models, meaning the models score so well that the tests reveal little. It also says those tests mostly cover short, single tasks. A model can do well on each task alone yet fail when it must put the tasks in order, follow a branching path, or fix errors caused by its own actions.

CyScenarioBench is built from real cyber incidents. Irregular says it studied those incidents, pulled out the attackers' decisions and constraints, and turned them into "attack trees", which are maps of the steps and conditions in an attack. It then built new, unseen environments from them. These run in containers, which are small, isolated virtual computers. They include defensive monitoring tools, simulated employees who react to social engineering (tricking people into giving access), and a test where the model must find passwords in simulated meeting recordings.

The test has three levels: single tasks, multi-step paths, and full campaigns. Irregular says it records failures such as losing track of context, choosing the wrong branch, and reaching dead ends. It says it also tracks how much the models help people with little skill.

Irregular says the test set is private to keep models from having seen it. It says some frontier models already use the benchmark. The post does not name them or give results.

What was done about it

This is a description of a testing method, not a response to an incident. Irregular lists its own limits. It says simulations cannot fully match real systems and people. It also says its way of measuring long-run error buildup is coarse. The test does not yet cover several AI models working together. Irregular says future versions will address these gaps.

Why it matters

Irregular says AI models are starting to influence real security incidents. It argues that testing full attack workflows shows more about real risk than testing isolated tasks. It points in particular to models that could guide people with little skill through an attack. The post gives no test results, so it does not say how capable any model is. Details are in the original report.