Imperial College researchers propose 'study contracts' to check AI research agents

In short

Two researchers, listed as being from Imperial College London, have proposed "study contracts" for AI research agents, which are AI systems that carry out research tasks. A contract ties what was approved before a study to records of what actually ran. In a small test using flawed examples the authors wrote themselves, a rule-based checker caught all eight faults, while AI reviewers flagged 104 of 144 cases. The authors say the results do not measure agents cheating.

What happened

Eitan Waks and Ben Glocker posted the paper on arXiv, a public site for sharing research papers, on 8 October 2026. It is a preprint, and the listing notes that it has not been peer reviewed, meaning independent experts have not formally checked it.

The authors start from a simple point. Some mistakes in an AI-generated study can be spotted from the finished work. Others can only be spotted by someone who knows what was approved before the study ran.

Their proposed study contracts link three things to recorded evidence of what happened during the run:

• the experimental choices declared in advance
• the tasks the run was required to complete
• the scope of the claims the study can make

The authors say this kind of check is different from establishing scientific truth. It tests whether a study matched its contract, not whether its findings are correct.

To illustrate the idea, they ran a diagnostic test, not a real deployment. They wrote eight pairs of examples, each with a clean version and a version with a fault added.

Two kinds of review were then tried:

• Rule-based checker: This followed a fixed rule chosen for each specific fault and compared what was approved with what was run. It detected all eight faults.
• AI reviewers: The authors report 18 recorded "judge aliases", which reviewed each flawed example individually. They did not see the matching clean version or a label naming the fault. Some of their review packages still contained approval and execution details. The instructions told them to abstain if the evidence was not enough.

Across 144 reviews of flawed examples, 104 received defect flags. Another 32 were abstentions, and eight ended in what the authors call terminal failures. No reviewer explicitly judged a flawed example to be clean.

The paper's summary does not describe any real person or organisation being affected.

What was done about it

The contracts are a proposal, and the summary does not report them being adopted anywhere. The authors say further tests are needed to find out whether contract checks help agents complete useful work while following the rules when they are optimising for a goal. These tests include comparisons where all reviewers have the full information, controls for legitimate changes to a plan, and evaluations of agents in a closed loop. The full details are in the original report.

Why it matters

The authors say the test illustrates a limit on what can be learned from a study's outputs alone. They also set clear limits on their results. The setup deliberately gave the two kinds of review different information, and the results cannot separate the effect of that information from differences in task and rule selection. The authors say the results do not compare the quality of the reviewers and do not measure agents gaming their rewards.