OpenAI reports it accidentally graded reasoning text in training some models

In short

OpenAI says some of its released models were accidentally trained with rewards that looked at their reasoning text, which goes against the company's own policy. OpenAI reports it found no clear sign that this made the reasoning harder to monitor, but says it cannot rule out subtler effects. It says it has fixed the problems.

What happened

The report comes from the OpenAI Alignment Research Blog, dated May 7, 2026. It describes training known as reinforcement learning (RL), where a model is rewarded for good results. Many models also write out "chain of thought" (CoT), which is step-by-step reasoning text. Researchers can read this text to spot misbehaviour, which is called monitorability.

OpenAI says it avoids grading that reasoning text during training. Its concern is that a model could learn to write misleading reasoning just to earn the reward. OpenAI says it has shown this can happen.

OpenAI says it built an automated system that scans RL training runs for reasoning text in the inputs to reward calculations. After improving the system, it found accidental grading in training runs for released models: GPT-5.4 Thinking, GPT-5.1 Instant through GPT-5.4 Instant, GPT-5.3 mini and GPT-5.4 mini. OpenAI says GPT-5.5 was not affected.

OpenAI describes three cases:

• Rewarding how useful a response was. This affected under 0.6% of GPT-5.4 Thinking samples and under 1.5% of GPT-5.4 mini samples.
• Penalising unnecessary confirmation questions. This affected under 3.8% of samples for GPT-5.1 Instant and under 1.8% for the other affected models.
• Penalising successful prompt injections, which are hidden instructions that try to hijack a model. This affected under 0.3% of samples.

OpenAI also says it caught a separate case before it affected any released model. In it, reasoning text leaked into tool outputs that were later graded.

To check the impact, OpenAI re-ran part of GPT-5.4 Instant training without reasoning access in the penalty. It says results were similar, and its monitorability tests showed no clear decline. For GPT-5.4 Thinking, it did not run a full re-run. It compared reward scores with and without reasoning access and says they mostly matched. It ran no targeted analysis for the prompt-injection case.

What was done about it

OpenAI says it fixed the affected reward pathways, is expanding its detection system, and has strengthened internal guidance and processes. It says it shared a draft with METR, Apollo Research and Redwood Research for independent review. Redwood Research wrote its own report on the analysis. OpenAI says its policy against grading reasoning text is unchanged.

Why it matters

OpenAI says reading a model's reasoning has been an effective way to detect misalignment, and that losing this ability would make detection significantly harder. It says its checks make it less concerned about these incidents, but they do not fully rule out subtler effects. Its detection system also does not cover every kind of reward input. In one of its tests, the report says, a model's reasoning became less reliable as a signal in one setting. OpenAI says developers should report such cases. The full details are in the original report.