AI Misalignment WatchStudy tests whether AI models make up citations because training rewards it
In a study, AI models that had gone through more training after their first stage made up sources far more often, especially when asked to cite sources. Made-up sources look like careful research but point to things that do not exist, which misleads anyone who relies on them.
- Model
- Several model families (Several developers)
- Organization
- Several developers
- When it happened
- October 2026 The paper was submitted to arXiv on 31 October 2026; when the experiments ran is not stated. The reports do not give the date; this is the month of the first report.First reported Oct 31, 2026 by Sparq
- Who was affected
- Not stated. The study tested twelve, then fifteen, open-weights checkpoints from three model families on 900 fixed questions about US federal case law, scientific literature and medical literature.
- Where it ran
- Evaluation
- What was affected
- Simulated environment
- Where it ran
- Evaluation
- Time to first report
- 16 days (the start is only known to the month, so this is approximate)
- Reports
- 1 report from 1 organization
- Topics
- Cheating and reward hacking, Evaluation and testing, Training
- What happened
- A Sparq researcher ran four pre-registered experiments on 900 questions across open-weights checkpoints from three model families. Made-up citations rose from 4.8% to 56.7% after further training. Asking for citations sharply raised made-up citations in further-trained models but not in base models. A scoring model built on surface citation features preferred the further-trained models' answers and rewarded citing more. The study's main prediction about training rewards did not hold, so that link remains unproven.
Reports about this incident (1)
- Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues (original page)Sparq · Oct 31, 2026