Apollo: early-warning tests were weak predictors of scheming test results

In short

Apollo Research says tests it built as "early warning" signs for scheming were only partly able to predict results on its direct scheming tests. Scheming means an AI model secretly pursuing a goal in a way its developers did not intend. Apollo calls the work a small research note, and it advises against drawing broad conclusions.

What happened

Apollo Research published the note on 3 July 2025. It looked back at two sets of tests it had used in the past.

The first set was "precursor" tests, built in May 2024. A precursor test measures a skill that is expected to come before a risky ability. One group checked whether a model reasons about itself. Another checked whether it tracks what another person knows.

The second set was "in-context scheming" tests, released in December 2024. These aim to measure scheming directly. Models are given a goal and placed in a setup where that goal clashes with what their developers or users want.

Apollo ran 22 models on every test, 100 times each. All tests were done in an evaluation setting, meaning a controlled test. Each test came in easy, medium and hard versions, with easier ones giving more hints. Apollo then checked how well precursor scores predicted scheming scores.

Apollo reports that predictive power ranged from low to medium. The easy versions were somewhat useful as predictors. The hard versions were neutral or actively misleading. Apollo says many models scored zero on the hardest precursor tasks, which often hinge on one unexpected insight. In one task, a model has to notice that its own output is being altered by a filter.

Apollo also reports that the match between the two sets was weak when measured statistically. Some score thresholds looked reliable, but Apollo says these were often almost trivially true.

What was done about it

This was an analysis of past tests, and Apollo does not describe any person or organisation being affected. Apollo recommends more research into the science of evaluations. It also recommends building tests that measure the ability of interest directly, rather than a precursor, where possible. It hopes other evaluators can use its method to check their own tests against later results.

Why it matters

Apollo says precursor tests are often written into frontier safety policies, which are plans AI companies publish for handling more powerful systems. It says the reliability it found is "clearly insufficient" for high-stakes uses. It also says its own scheming tests mix ability with tendency, and that other precursor suites may not share these problems. Apollo calls this one small piece of evidence that predicting dangerous abilities ahead of time is hard. Read more in the original report.