← Reports and Research

Research report AI company Apple

Topics: Alignment research, Cheating and reward hacking, Training

Apple paper maps the theoretical limits of aligning language models

In short

Apple Machine Learning Research has published a paper on the theoretical limits of "alignment," which means steering a language model's answers toward what people prefer. The authors say one common method comes close to the best possible result, while two popular training methods fall well short. They also say combining several reward models helps reduce "reward hacking."

What happened

The paper is by Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald and Federico Danieli. Apple lists it as a NeurIPS conference paper, published in November 2026. This is a research study, not a report of a model misbehaving in the real world.

The authors start from a gap they say exists. Two methods are widely used to align models. One is reinforcement learning, a training method that rewards good outputs. It is held in check by a "KL-divergence constraint," a limit on how far the model may drift from its original behavior. The other is best-of-N, where the model writes N answers and a scoring system picks the highest-scoring one. The authors say the basic limits of how much reward can be gained under a fixed KL budget "remain poorly understood."

They work out the maximum possible gain in expected reward for a given budget. They say the result is governed by a Jeffreys divergence term, rather than the square root of KL used in earlier analyses. They also describe a practical estimator that predicts achievable gains from samples of the base model alone.

They then look at "proxy" rewards. A proxy reward is an imperfect stand-in for what people actually want. The gap between ideal and proxy results is what the paper calls reward hacking, where a model scores well without doing what was intended. The authors say this gap grows as reward errors get larger and as the KL penalty factor gets smaller. They also prove that reward ensembling, which means combining several reward models, reduces reward hacking.

Finally, they test two tasks: safety and summarization. They say best-of-N comes close to the theoretical limit. They say two reinforcement learning methods, PPO and GRPO, "remain substantially suboptimal."

What was done about it

Nothing was done in response, because the paper does not describe an incident. The authors say their results suggest that "algorithmic improvements are needed" to reach optimal alignment without high inference costs. Inference cost is the computing needed each time a model produces an answer.

Why it matters

The authors say their results help explain several patterns seen in earlier alignment research. They also say the work gives a theoretical basis for reward ensembling, a technique already used in practice. The source page is a summary only. It does not give detailed experiment results, and it does not say whether other researchers have reviewed the findings. Read more in the original report.

This is a summary in everyday words, written by AI Misalignment Watch from the source above. If you spot a mistake, the original always wins.