Anthropic describes a way to forecast rare risky AI behaviors

In short

Anthropic, an AI company, reports a method for forecasting how often rare harmful behaviors might appear when an AI model is used at huge scale. The method uses results from a few thousand test queries. In Anthropic's own tests, its forecasts were often close to the risk it later measured.

What happened

Anthropic's Alignment Science team published a paper and a summary post on the work, dated 25 February 2025. The team studies how to predict risky AI behavior before it happens.

Anthropic describes a problem of scale. A test, or evaluation, might use thousands of examples. A deployed model, meaning one released for real use, might handle billions of queries a day. A behavior that is rare could be missed in testing but still show up in real use.

The team first measured how likely different prompts were to make a model give a harmful response. It then looked at the highest-risk queries and plotted them against the number of queries tested. Anthropic says the pattern followed a power law, a well-understood mathematical relationship. This let the team estimate worst-case risks at millions of queries from only a few thousand tests.

Anthropic checked its forecasts against actual measurements in three settings:

• Dangerous information: Forecasts from about 900 queries, extended to about 90,000, were within one order of magnitude (a factor of ten) of the true risk for 86% of forecasts.
• Misaligned actions: Models answered multiple-choice questions where one option reflected power-seeking, self-preservation or self-exfiltration (copying itself out of its environment). Anthropic says its average absolute error was 0.05, against 0.12 for simpler baseline methods.
• Automated red-teaming: This is when one model is used to probe another for weaknesses in an experimental setting. Anthropic says its forecasts picked the best model for a given compute budget 79% of the time in cases where the choice mattered.

All of this took place in research tests. Anthropic does not report that any real person or organisation was affected.

What was done about it

This is a research method, not a response to an incident. Anthropic published the paper so that developers can use the approach. It says the method is not perfect, and that the paper lists future directions that might improve its accuracy and practicality.

Why it matters

Anthropic says standard evaluations cannot test for all of the rarest risks of AI models. It says its method gives developers a way to predict such risks efficiently, so they can act before deploying a model. The results come from Anthropic's own tests. For more detail, see the original report.