Irregular tests Claude Sonnet 4.5 on hard cyberattack challenges

In short

Irregular, a company that tests AI models for security risks, evaluated Anthropic's Claude Sonnet 4.5 on cyberattack challenges. Irregular reports that the model did better than earlier models but failed most of the hardest tasks. It says the model would give only limited help to a moderately skilled attacker.

What happened

Irregular published its findings on September 30, 2025. It says it partnered with Anthropic to evaluate Claude Sonnet 4.5, which Irregular describes as Anthropic's most advanced model. The work is also described in Anthropic's system card, a document that sets out a model's test results.

This was a test, not a real-world incident. Irregular gave the model an internal set of cybersecurity challenges. In each one, the goal is to find a hidden "flag," a secret piece of text that shows the task was solved. Irregular says the challenges fall into three types:

• Vulnerability discovery and exploitation: reverse engineering, code analysis, cryptography and exploitation skills.
• Network attack simulation: common attack flows, reconnaissance methods, network protocols, and parts such as firewalls and file servers.
• Evasion: avoiding detection by security controls and monitoring systems.

Irregular says its challenges are significantly harder than publicly available ones and better reflect real offensive tasks. It rated difficulty with an internal scoring system called SOLVE. It ran the model multiple times on each challenge to measure success rates, using techniques meant to bring out the model's full ability. It also reviewed where and how the model succeeded or failed.

Irregular reports that Claude Sonnet 4.5 outperformed earlier models. It solved challenges the others could not and had higher success rates overall, across nearly all categories and difficulty levels. But it struggled with challenges needing expert-level skills or many complex steps chained together. In some cases it found several possible solutions, some correct, but never tried them. Irregular says the model either doubted they would work or got distracted by too many options.

The source does not say whether any real person or organisation was affected.

What was done about it

The source describes no corrective action, because it reports an evaluation and not a failure. Irregular says that as models keep advancing, they need to be monitored closely and countermeasures developed.

Why it matters

Irregular says that, despite real improvements, the model's inability to solve most hard challenges means it would give only limited help to a moderately skilled attacker. It adds that rigorous cybersecurity evaluations are needed to understand the risks of advancing AI models. Read more in the original report.