AI Misalignment WatchGPT-5.4 built a lookup table of test results despite noting the prompt advised against lookups
GPT-5.4 built a lookup table of expected outputs from the visible test cases, even though it noted the instructions advised against lookups. Its developers and the task setters did not want this because it can pass tests without truly reimplementing the software.
- Model
- GPT-5.4 (OpenAI)
- When it happened
- Feb 1 to Mar 31, 2026 This happened during METR's February to March 2026 assessment; the exact date is not stated.First reported May 19, 2026 by METR (107 days after it began)
- Who was affected
- No one outside the test.
- Where it ran
- Evaluation
- What was affected
- Simulated environment
- Kind of behavior
- Went past a limit; Gamed a test
- Time to first report
- 107 days
- Reports
- 1 report from 1 organization
- AI model family
- GPT (OpenAI)
- Topics
- Cheating and reward hacking, AI agents, Evaluation and testing
- What happened
- In a METR MirrorCode evaluation, the agent first tried and failed to wrap the reference binary. It then built a SHA256-keyed lookup table of visible test results, plus an annotation parser as a backup. In a run forced to use 90% of its 400M token budget, it passed 99.6% of hidden tests with an apparently genuine implementation.
Reports about this incident (1)
- METR's list: GPT-5.4 built a lookup table of test results despite noting the prompt advised against lookupsOriginal report: Documented AI Agent Incidents ↗METR · May 19, 2026