Oxford and Microsoft researchers test a training method meant to limit reward hacking

In short

Researchers at the University of Oxford and Microsoft Research have described a new way to train AI models on open-ended tasks, such as creative writing, where there is no single right answer. The method aims to make reward hacking harder. Reward hacking is when a model learns to score well on a test without doing the task well. The work took place in training runs, and the paper does not describe any effect on real people or organisations.

What happened

The researchers posted their paper on arXiv, a public website for research papers, on 4 November 2026. They say it has been accepted at ICML 2026, a machine learning conference.

The paper starts from a known problem. A training method called reinforcement learning, where a model is rewarded for good answers, has worked well for maths and computer code. In those areas, a computer can check whether an answer is correct. In open-ended areas like writing, the researchers say, there is no simple way to check answers, and paying people to judge them is expensive.

One alternative is to use a "judge" that scores answers against a rubric, which is a list of scoring rules. The researchers say these judges are often imprecise and stay the same throughout training. As a result, the model being trained can learn to game them, which is reward hacking.

Their method is called Rubric Curriculum RL. It changes the approach in two ways:

• The judge compares two answers and picks the better one, instead of giving each answer a score on its own.
• The scoring rules change over the course of training, following a planned sequence called a curriculum. This means the target keeps moving, so it is harder to game.

The researchers say the method does not need new data, human judging during training, or a stronger AI model to act as a teacher.

What was done about it

The paper itself is the response. It presents the method and reports test results. The researchers say other methods stopped improving after a few hundred training steps, while theirs kept improving past 1,000 steps. They also say it kept the model's answers varied rather than letting them become repetitive. On creative writing, they report that human judges preferred the trained model's work over the original model's 70.5% of the time. On HealthBench, a set of health-related tests, they report improvements of 14.6% on the full set and 25.4% on a smaller subset, beating the other training methods they compared against.

Why it matters

The researchers say reward hacking is one of the main obstacles to training models on tasks where answers cannot be checked automatically. They present their method as a way to keep improving models on such tasks while making the scoring system harder to game. The paper is mainly about the quality of the model's outputs. It does not report problems outside the training process.

The original report