Anthropic tests an AI constitution written with input from the public

In short

Anthropic, an AI company, and the Collective Intelligence Project asked about 1,000 Americans to help write the rules for an AI chatbot. Anthropic then trained a model on those rules. Anthropic reports that it matched a standard model on helpfulness and harmlessness, and scored lower on one bias test.

What happened

Anthropic published the original report on 17 October 2023. It describes a research experiment, not a problem in a live product.

Anthropic uses a method called Constitutional AI. A "constitution" is a written list of principles that a model is trained to follow. Anthropic's chatbot Claude normally uses a constitution written by Anthropic staff. Anthropic says this gives developers an outsized role in choosing an AI's values.

To test an alternative, Anthropic and the Collective Intelligence Project used Polis, an open-source platform for online group discussion. About 1,000 people, chosen to represent U.S. adults by age, gender, income and geography, voted on rules or proposed their own. They contributed 1,127 statements and cast 38,252 votes. Polis found two separate opinion groups, though most statements drew high agreement.

Anthropic kept statements that reached a threshold of agreement within both groups. It says about half the concepts and values overlapped with its own constitution. The public version put more weight on objectivity, impartiality and accessibility, and more on encouraging good behavior than avoiding bad behavior.

Anthropic then trained two small models, one on each constitution, and compared them. It reports:

• Both scored about the same on standard language and math tests.
• People rated the public-input model as helpful and harmless as the other models, with no significant differences.
• The public-input model scored as less biased across nine social dimensions on a test called BBQ.
• Both models' answers matched the views of self-described liberals more than conservatives. The differences were small but statistically significant.

What was done about it

Nothing needed fixing, but Anthropic listed the limits of its work. It says many subjective choices were involved, including how to moderate statements, remove duplicates and rewrite public statements into training format. It says it is unsure it struck the right balance on duplicates. It also used the same set of training prompts for both models, which it calls "likely a mistake" because some public principles may not suit those prompts. Early versions of both models were overly cautious and unhelpful, which Anthropic fixed by changing how training data was weighted.

Why it matters

Anthropic says it expects questions about the democratic legitimacy of AI to become more prominent. It calls its work "very preliminary and imperfect" and says the sample is U.S.-only and not globally representative. It also says it saw only a small, clear difference in bias on the tests it chose, and that training with Constitutional AI was harder than expected. It invites feedback and says it hopes others can build on the work.