Anthropic studies simple ways to make AI assistants helpful, honest and harmless

In short

Anthropic reports early research on how to build a text-based assistant that is helpful, honest and harmless. The page, dated December 1, 2021, describes tests of simple methods, including prompting, and of several training approaches. Anthropic says one approach, ranked preference modeling, performed much better than imitation learning.

What happened

This is a research report, not an account of an AI system causing a problem. Anthropic describes it as an "initial foray" into building a general-purpose assistant that is aligned with human values. By "aligned," it means the assistant is helpful, honest and harmless.

The work was done as research experiments. The page does not say it involved a real deployment, and it does not say any person or organisation was affected. It does not name the specific models tested.

Anthropic first studied simple baseline techniques and evaluations, such as prompting. Prompting means giving a model written instructions or examples before it answers. Anthropic says the benefits of these modest steps grew with model size. It also says they carried over to a variety of alignment evaluations and did not reduce the performance of large models.

Anthropic then compared how three training objectives behave as models get larger:

• Imitation learning, where a model learns to copy examples of good behaviour.
• Binary discrimination, where a model learns to tell good answers from bad ones.
• Ranked preference modeling, where a model learns to rank answers from better to worse.

Anthropic reports that ranked preference modeling performed much better than imitation learning, and often improved more with model size. It says binary discrimination typically performed and scaled very similarly to imitation learning.

Finally, Anthropic says it studied a "preference model pre-training" stage. The goal was to improve sample efficiency when fine-tuning on human preferences. Sample efficiency means getting good results from less training data. The page's summary does not give the outcome of this last step.

What was done about it

The page does not describe a problem that needed a response. Anthropic published the findings as a paper, which the page links as "Read Paper." The page itself gives only the abstract, so details of the methods and results are in the paper rather than here.

Why it matters

Anthropic presents the work as a starting point for aligning a general-purpose text assistant with human values. Its stated finding is that some training methods scale better than others as models grow, and that simple steps such as prompting can help without hurting large-model performance. The page gives no further detail on how these results were measured. Readers can find the full account in the original report.