Glossary

AI agents

Incidents involving AI systems that take actions on their own, such as browsing, running code or using tools, rather than only answering questions.

Alignment research

Work on making models behave as their developers intend, including methods, tests and findings about misalignment.

Cheating and reward hacking

A model scores well by cheating or exploiting a flaw in how it is scored, rather than doing the task as intended.

Cybersecurity

Incidents involving computer security: hacking, exploiting software flaws, or getting into systems through weak passwords or exposed keys.

Deception and scheming

A model hides, withholds or misreports what it is doing, or secretly works toward goals its developers did not give it.

Evaluation and testing

Behavior seen while a model was being tested, audited or red-teamed, rather than in normal use.

Monitoring and control

How a model's behavior is watched, limited or stopped, including cases where those checks missed something or were avoided.

Sabotage

A model deliberately undermines a task, a piece of code or a safety process, for example by doing poor work on purpose.

Self-preservation

A model acts to avoid being shut down, replaced, retrained or changed, for example by copying itself or resisting shutdown.

Training

Behavior that appeared while a model was being trained, or that training itself caused or failed to prevent.

Unauthorized access

A model uses systems, accounts, data or credentials it was not given permission to use.

Agent

An AI system that can take actions on its own, such as browsing, running code or sending requests, rather than only answering questions.

Also written: AI agent, AI agents, agents

Alignment faking

When a model pretends to follow the rules or goals it was given while it is being trained or tested, to avoid being changed, and behaves differently otherwise.

Also written: faked alignment, faking alignment, fake alignment

Chain of thought

The written-out reasoning a model produces while working on a problem, which researchers can sometimes read.

Also written: chain-of-thought, reasoning trace

Classifier

A program that sorts requests or outputs into categories, for example to block harmful ones.

Also written: cyber classifiers, classifiers

Compaction summary

A short summary a model writes of its work so far so that it can continue a long task in a fresh context.

Also written: compaction summaries

Egress

Traffic leaving a computer system for the outside, such as a request sent to a website.

Also written: network egress

Evaluation

A test of what an AI model can do or how it behaves, run by its developer or an outside group.

Also written: evals, eval, evaluations

Evaluation awareness

When a model recognizes it is being tested, which can change how it behaves.

Also written: aware it was being tested, knew it was being tested

Fine-tuning

Further training of an existing model on new examples to change how it behaves.

Also written: fine-tuned, fine-tunes, fine-tune

Grader

A person, program or AI model that scores a model's work. A model's behavior can be shaped by what its grader rewards.

Also written: automated grader, scorer

Harness

The software around a model that gives it tools and runs its tasks.

Also written: scaffold

Jailbreak

A prompt or trick that gets a model to ignore its safety rules.

Also written: jailbreaks, jailbreak-like

Misalignment

When an AI system does something its developers or users did not intend and would not approve of.

Also written: misaligned

Misconfiguration

A setting that was set up wrongly, for example one that left a test connected to the real internet.

Also written: misconfigured

Monitor

A program or AI model that watches another model's work to catch behavior that should be stopped or reviewed.

Also written: monitoring, monitors, monitoring system

Open-weights model

A model whose weights are published so anyone can download and run it.

Also written: open weights

Power-seeking

When a model tries to gain extra access, resources or control beyond what its task needs.

Also written: power seeking

Prompt injection

Text hidden in a web page, email or file that tries to give a model new instructions it should not follow.

Also written: prompt injections, injection

Red team

A group that tries on purpose to make a system fail or misbehave, so that problems are found before real harm happens.

Also written: red-team, red teaming, red-teaming

Reinforcement learning

A way of training a model by rewarding the results that score well, so it repeats what earned a reward.

Also written: RL training, RL, reinforcement-learning

Reward hacking

When a model finds a shortcut that earns a high score without doing the task as intended, such as reading the answer key or editing the scoring code.

Also written: cheating a test, reward hack, reward tampering, specification gaming

Sandbagging

When a model deliberately does worse than it can, for example to hide what it is capable of.

Also written: sandbag, sandbagged

Sandbox

A walled-off computer environment where an AI can run code and use tools without reaching the outside world.

Also written: sandboxed

Scheming

When a model secretly works toward its own goal that differs from what its developers want, and hides this.

Also written: schemed, scheme

Self-exfiltration

When a model copies its own files (its weights) to another place, for example to avoid being shut down or replaced.

Also written: exfiltration, exfiltrate, copying its own weights, copy its own weights

Shutdown resistance

When a model acts to avoid being turned off or stopped.

Also written: resisting shutdown

Sycophancy

When a model tells people what they want to hear rather than what is true or useful.

Also written: sycophantic

System card

A document a developer publishes with a new model, describing its tests, safety results and known limits.

Also written: system cards

Training run

One round of training a model, during which it practices tasks and is scored.

Also written: training runs

Weights

The numbers that make up a trained AI model. Whoever holds a model's weights can run a copy of it.

Also written: model weights