Researchers test a check for when picking AI's top-scored answer backfires

In short

Researchers led by City University of Hong Kong have described a check for when picking the top-scored AI answer backfires. In their tests on simulated data and real sets of answers, these failures were rare, and all of them occurred in code generation. The paper's summary does not report harm to any real person or organisation.

What happened

The paper, posted on arXiv, a website for sharing research papers, looks at a method called Best-of-N. An AI model produces a number of answers, called N. A scoring system then rates them, and the highest-scoring answer is returned.

The scoring system is only a stand-in for true quality, so it can make mistakes. The paper's title calls the resulting problem "reward hacking". Here, that means picking answers that score highly but are less correct. The authors say that a larger N tests the very top end of the scores, known as the tail, more than the scorer's average ability to rank answers.

Using mathematics, the authors report that results can get worse as N grows only if something specific happens in the tail. Among the top-scoring answers, true quality has to start falling as scores rise. Where this turn happens affects the value of N at which results start to decline.

The authors also studied how different patterns of scoring errors change the results:

• Gaussian errors, which rarely stray far from the true value: results keep improving as N grows.
• Laplace errors, where large mistakes are somewhat more common: results level off.
• Heavy-tailed errors, where very large mistakes are more common still: results fall back to the average level.

They add that if the scorer's errors vary most on low-quality answers, even Gaussian errors can make Best-of-N worse than choosing at random.

The authors then ran two kinds of tests. Simulated data agreed with these patterns. They also used real sets of answers from several subject areas where correctness can be checked, with different AI models and scoring systems. The authors say failures in the tail were rare in these sets, and all of them occurred in code generation.

What was done about it

The authors propose two tools:

• A check called tail calibration: this measures how well top scores match true quality, using data kept aside from the main analysis. The authors say it reliably predicts where limiting N helps.
• A limit on N: the authors say this limit is competitive with an existing comparison method called Soft-BoN.

The check predicted where the limit would help about as well as a measure of ranking accuracy did. The authors say it does not replace ranking measures.

Why it matters

The authors describe the check as an audit that shows which values of N are at risk. They say its results do not change if the scores are rescaled in a way that keeps their order. The summary of the original report does not give failure rates or name the subject areas tested, other than code generation.