Irregular outlines best practices for building AI evaluations and test suites

In short

Irregular, a company that evaluates AI systems, has published guidance on how to build good tests of AI risk. The October 2024 post lists qualities that single tests and groups of tests should have. No AI model was tested. This is advice, not a report of any model's behaviour.

What happened

Irregular published the post on October 2, 2024. It is the second part of a series on designing "evaluations," which are tests of what an AI system can do. A group of them is called an "evaluation suite."

The first part covered the steps to take before building any test. These include deciding what decision the results will feed into and setting thresholds, meaning the level of ability at which a risk becomes a concern.

This second part describes qualities Irregular says most single tests should have:

• Realistic. Tests should ask a model to carry out a real task, not answer multiple-choice questions. Irregular says some hacking competitions use riddle-like puzzles that are hard to connect to real risks.
• Clear difficulty. There should be a known solution. Irregular says there should also be high confidence that no easier solution exists.
• Not in the training data. If a model has seen a test before, its result may reflect memorised answers rather than real ability. Irregular calls this contamination. It says this is hard to rule out in practice.
• Clear focus. The skill being tested should be stated precisely. "Writing malware" and "writing code that evades the three most common EDR products" would give different results.
• High signal density. Checkpoints and logs of a model's work can show more than a simple pass or fail.
• Fitting scoring. The scoring method should suit the decision the test serves.

For whole suites, Irregular says coverage should vary along factors that could change performance, such as vulnerability type or programming language. It says even irrelevant information can change a model's success rate. Difficulty should be chosen deliberately, and suite scores should make sense to the intended user, who may not be an expert. Tests should also overlap, so one lucky or unlucky result does not mislead.

What was done about it

Nothing was reported as needing a fix. Irregular says it plans to cover further factors in the next part of the series. It also says general good practice, such as thorough quality checks, applies but is not covered here.

Why it matters

Irregular says that without these qualities, a test might not produce results anyone can act on, or might not contain enough meaningful information. The post notes that there are rare exceptions to its guidance. It also says some of these goals, such as ruling out training-data overlap, are difficult to confirm. Read more in the original report.