Anthropic tests whether interpretability tools help explain odd AI behavior

In short

Anthropic researchers built a tool called CHIVE that finds unexpected behavior in AI language models and tests what causes it. Anthropic reports that tools which read a model's internal activity did not help agents predict the results any better than reading the conversation alone. Models trained on the data did get better at predicting their own behavior.

What happened

Anthropic's Alignment Science team published the original report on its research blog. It describes research work. No real person or organisation was affected, and the source does not describe a real-world deployment.

CHIVE works in four steps. It samples 30 responses from a target model for each prompt. A model then flags unexpected behaviors. An agent runs 5 to 15 experiments that edit the prompt and measure how often the behavior changes. A separate judge scores how well the experiments support the explanation.

The researchers then tested three interpretability tools. Interpretability tools try to show what is happening inside a model. All three read the model's activations, which are the numbers it computes internally. The tools were activation oracles, natural-language autoencoders and sparse autoencoders.

A predictor agent, Claude Opus 4.8 in the main experiments, judged whether a proposed prompt edit would change a behavior. Anthropic reports that none of the three tools beat an agent that only read the transcript. This held across two target models, three predictor model families and several settings.

The authors say the tools sometimes helped. They say those wins were offset by cases where the output missed the key information or misled the predictor. They also say the tool outputs almost never stated the cause-and-effect link directly.

What was done about it

This was a research study, not a response to an incident. The authors also used the CHIVE data to train two models, Qwen3-8B and Qwen3.5-397B-A17B, to predict how prompt edits would change their own behavior. Anthropic reports that both improved substantially over their base versions. This held on a "hint" test that was not part of training and on held-out investigations. Training models to write open-ended explanations of their own behavior gave weaker, mixed results.

Why it matters

The authors say their test is only a proxy for how interpretability tools are used in practice. The behaviors it found were fairly simple, and real uses often ask what a model "believed" rather than what would change its behavior. They say the results do not invalidate those uses, but do not validate them either. Until that changes, they think causal claims based on these tools should be treated as suggestive evidence. They also suggest that future investigations separate what a transcript already shows from what only the tools reveal.