Topics: Deception and scheming, Evaluation and testing, Training
Apollo Research urges outside checks of AI training runs for scheming
In short
Apollo Research, an AI safety group, argues that outside groups should review how advanced AI models are trained, not just the finished model. It says "scheming" may be easier to spot partway through training. The post is an argument and a proposal. It does not report any model misbehaving.
What happened
Apollo Research published the original report on its website. It calls the idea a "Training-Run Assessment," or TRA. This is a close look at the steps that lead up to a model's release. That includes saved snapshots of the model during training, which are called checkpoints.
Apollo defines scheming as an AI secretly pursuing goals its developers did not intend, while hiding those goals or its abilities. Apollo says tests run only on the final model assume scheming can be reliably found at that stage. It says that may not be true. A trait might show up midway through training and then become hidden by the end. Apollo says it is "very uncertain," but thinks this is a likely scenario.
Apollo gives three reasons for using outside reviewers:
- Developers may have little reason to look for scheming that is hard to see. Apollo says its impression is that safety work during model building is already rushed.
- A developer's own safety claims are hard for others to trust. An independent group could review private material and report only its conclusion.
- Outside reviewers could pool expertise and share it across several developers.
Apollo describes three kinds of review, with deeper access at each step:
- Checkpoint evaluations: testing intermediate versions of the model, with no access to training data.
- Data evaluations: examining what goes into training, such as datasets and practice environments.
- Process reviews: developers record what they saw during training and how they responded.
Apollo says the main obstacle is protecting trade secrets. It points to financial auditors and chip makers as examples of outsiders who already handle sensitive information. It also says no one has yet seen scheming emerge in real-world models, so it is not clear how best to test for it.
What was done about it
Nothing was being fixed, because no incident is described. Apollo says it intends to conduct third-party Training-Run Assessments in the future. It urges developers to start with checkpoint evaluations that the outside group designs and the developer runs. It also suggests small pilots using narrow slices of data or older training runs. It asks policy makers to start thinking about how such reviews could work.
Why it matters
Apollo says that testing only final models may miss problems visible earlier in training. It argues that outside checks would give the public and policy makers more reason to trust developers' safety claims. The post is Apollo's own position. It also says scheming might not be detectable at any point in training, in which case neither approach would be enough.
This is a summary in everyday words, written by AI Misalignment Watch from the source above. If you spot a mistake, the original always wins.