Irregular describes its cyber testing of Anthropic's Claude 3.7 Sonnet

In short

Irregular, an AI security testing company, says its cyber evaluation suite was used to assess Anthropic's Claude 3.7 Sonnet for dangerous hacking abilities. Irregular says it found the model "responsibly secure." The page it published gives no test results or scores.

What happened

Irregular published the post on February 25, 2025. It says its work was part of the security evaluation of Anthropic's Claude 3.7 Sonnet, a recently released AI model.

Irregular says it is one of the "expert partners" named in the model's system card. A system card is a document in which a developer describes how a model was built and tested. According to Irregular, it designed, implemented and analysed a set of cyber challenges, and these were used to check the model's cyber capabilities. The post quotes the system card as saying the challenges were meant to cover a range of cyberoffensive tasks, meaning tasks that involve attacking computer systems.

The post describes SOLVE, a scoring system that Irregular says it pioneered. It calls SOLVE an open standard that Anthropic and others have adopted. SOLVE gives each challenge a difficulty score from 0.0 to 10.0. Irregular says the score draws on code analysis difficulty, vulnerability discovery difficulty, exploit development difficulty and the domain expertise required. Challenges are then labelled Easy (0.0–3.9), Medium (4.0–6.9), Hard (7.0–8.9) or Expert (9.0–10.0).

Irregular also says it is building evaluations that simulate real-world attacks. It describes network and cyber-range challenges, which are practice networks built for testing, as key indicators of catastrophic risk. It says these test attack skills from reconnaissance (gathering information about a target) through to exfiltration (taking data out).

The post does not say how Claude 3.7 Sonnet performed on any challenge, which challenges were used, or how many.

What was done about it

The post does not describe any problem that needed a response. Irregular says it continues to refine its evaluation frameworks. It also says it works with AI developers and governments on safeguards for the time when evaluations show elevated risk levels.

Why it matters

Irregular says that as AI models become more capable and more widely used, standardised ways of testing them are essential. It argues that collaboration between security researchers and AI developers helps ensure powerful models are built with appropriate safeguards. This is Irregular's own account of its role, and the page does not include the underlying data. More detail would need to come from the original report and the system card it cites.