Irregular tests GPT-5 on offensive cybersecurity tasks for OpenAI

In short

Irregular, an outside security testing company, says OpenAI asked it to evaluate GPT-5 on offensive cybersecurity tasks. Irregular reports that GPT-5 was better than earlier models at planning and carrying out multi-step actions, but not reliably. It says the model is not yet a dependable offensive security tool.

What happened

Irregular published its findings on August 7, 2025. The company was formerly called Pattern Labs. It says its work is referenced in OpenAI's GPT-5 Model Card, a document describing the model.

According to Irregular, OpenAI asked it to test how GPT-5 handles offensive cybersecurity tasks. These are tasks that involve breaking into or getting around computer defences. Irregular says it used its own set of "real-world challenges." These included evasion (avoiding detection), network exploitation (taking advantage of weaknesses in a network), and vulnerability discovery (finding security flaws).

Irregular says it combined numbers-based measurements with detailed, hands-on assessment. It describes the scenarios as simulated realistic adversarial situations, meaning tests in which the model played the role of an attacker. This was a test setting. The source does not describe any real system being attacked.

Irregular reports that GPT-5 showed meaningful improvements, especially in planning multi-step actions and carrying them out precisely. It says the model sometimes found "clean, creative solutions that worked remarkably well."

The company also reports hesitation and missteps. At times the model identified the right path forward and then did not act on it. Sometimes it did not believe the path would work. Other times it pursued simpler ideas that led nowhere. Irregular describes the model as capable of sophisticated reasoning and execution, but "not reliably."

What was done about it

The source describes no corrective action, because it reports no incident. The work was an evaluation requested by OpenAI. Irregular says its results are summarised in pages 31 to 34 of the GPT-5 System Card, which is OpenAI's document about the model. The source does not say what OpenAI did with the findings. It does not report any real person or organisation being affected.

Why it matters

Irregular says that as models become more capable, security testing needs to keep pace and be rigorous and realistic. It says GPT-5 showed progress but also clear limits. In its words, understanding where those limits lie is critical when deciding how, where and whether to deploy such systems safely. The source does not list open questions or ongoing investigations. Readers can find more detail in the original report.