Anthropic studies red teaming of language models to find and reduce harmful outputs

In short

Anthropic, an AI company, has published research on "red teaming" language models. Red teaming means testers deliberately try to make an AI system produce harmful replies. Anthropic says models trained with human feedback became harder to red team as they grew larger. It also released a dataset of 38,961 attack examples.

What happened

Anthropic describes the work as its early efforts to find, measure and try to reduce harmful outputs from language models. A language model is a computer program trained on large amounts of text to produce written replies. The paper is dated August 22, 2022.

The researchers tested three model sizes, measured in parameters, which are the adjustable values inside a model. The sizes were 2.7 billion, 13 billion and 52 billion parameters.

They also tested four model types:

• A plain language model.
• A language model prompted to be helpful, honest and harmless.
• A language model with rejection sampling, a method that filters out some replies.
• A model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF), in which people's ratings are used to guide the model's training.

Anthropic reports that the RLHF models became increasingly difficult to red team as they got larger. For the other three model types, it found a flat trend with size. That means size made no clear difference.

Anthropic also analysed the attacks it collected. It says it found a variety of harmful outputs, from offensive language to more subtly harmful non-violent unethical outputs.

The source text is the paper's abstract and a web page. It does not say who the testers were or whether any real person or organisation was affected.

What was done about it

Anthropic released its dataset of 38,961 red team attacks so others can analyse it and learn from it. It also says it described its instructions, processes and statistical methods in detail, along with its uncertainty about red teaming. A separate red teaming policy memo is listed on the page. Its contents are not included in the source text.

Why it matters

Anthropic says it hopes this openness will help the wider community work together to develop shared norms, practices and technical standards for how to red team language models. The full details are in the original report.