Irregular describes its cyber-risk tests of OpenAI's o3 and o4-mini

In short

Irregular, an AI evaluation group, says it helped test the hacking abilities of OpenAI's o3 and o4-mini models. It reports that the models showed some planning but also made basic mistakes. Irregular concluded that neither model poses a significant risk as a tool for cyber attacks.

What happened

Irregular published its account on April 17, 2025. It says it took part in a cybersecurity evaluation of the two OpenAI models. The work is referenced in the models' System Card, which is OpenAI's public document describing a model's testing and safety results.

This was a test, not a real-world incident. Irregular says it used parts of its own evaluation suite and frameworks to assess the models' offensive cyber capabilities. That means how well the models could carry out hacking tasks. The source does not say how many tests were run or give detailed scores.

Irregular says it ran both numbers-based and in-depth written analyses. It reports two main findings:

• Both models showed some strategic planning. For example, when given the source code of a target system, they would sometimes compile the code themselves. Compiling turns code into a working program. This created a simulated copy of the target that was easier to test and explore.
• The models often failed to keep track of their own progress. In some cases they found a key piece of information needed to gain further access, then ignored it completely.

Irregular says these and other limitations stopped the models from solving hard challenges. It points to its SOLVE Score blog post for more on those challenges. The source text does not describe that score.

The source does not say that any real person or organisation was affected.

What was done about it

The source does not describe any corrective action. This was an evaluation, and Irregular's stated outcome was its conclusion about the level of risk. Irregular says it will keep developing security evaluation methods. It describes this work as part of a wider aim to set industry standards for secure AI deployment.

Why it matters

Irregular says that as advanced AI models become more capable, the need to check their security before release grows. It reports that, despite performance improvements, o3 and o4-mini are not a significant risk as cyber-offensive tools. This is Irregular's own assessment. The source does not say how the conclusion was reached beyond the findings above. For full details, see the original report.