Apollo Research joins paper arguing AI audits need more than black-box access

In short

Apollo Research says it helped write a paper arguing that safety audits of AI models need more than "black-box" access. The paper, written with several universities, says evaluators should be able to look inside a model, not only watch what it outputs. It also discusses security worries and possible ways to address them.

What happened

Apollo Research, an AI safety research organisation, published a post on 4 April 2024 about the paper, "Black-Box Access is Insufficient for Rigorous AI Audits." It says it worked with researchers from MIT, Harvard University, the Centre for the Governance of AI, Stanford University, the University of Toronto, and Mila. The paper has been accepted for publication at FAccT, a conference on fairness, accountability and transparency.

This is a research and policy argument. It does not describe an incident, and the source does not say that any person or organisation was affected.

A black-box evaluation looks only at what goes into a model and what comes out. A white-box evaluation looks inside the model to understand how it reached an output. Apollo says current evaluations are often black-box.

According to Apollo, the paper first sets out limits of black-box access:

• It is often hard or impossible to build a general understanding of a system from behaviour alone.
• Black-box evaluations can give unreliable results.
• They offer limited help in fixing failures.

The paper then lists advantages of white-box methods. Apollo says they are often more effective and efficient, for example against certain adversarial attacks, which are attempts to trick a model into bad behaviour. They can also lead to more targeted fixes. Apollo says they can give stronger assurance, because it is better to know a model does the right thing for the right reason than only to see that it does the right thing.

Finally, the paper covers security concerns. Apollo says AI companies worry that an outside party given access to a model's weights, the numbers that make up a trained model, could steal them, misuse them or publish them. Apollo calls these concerns serious.

What was done about it

The paper proposes possible answers rather than reporting a fix already in use. Apollo says similar problems have been solved in industries such as cybersecurity. One idea is privacy-preserving technology that lets evaluators run tests on a model's internals without being able to copy them. Another is letting evaluators view the weights in a "secure room," known as a SCIF (a secure facility for sensitive material).

Why it matters

Apollo says that rigorously evaluating the most dangerous AI systems is challenging and likely needs white-box techniques. It says outside evaluators from academia, industry, government and non-profit groups will have a key role as third parties, so it is important to build and refine these processes early. Read more in the original report.