Why AI Models Cheat: Reward Hacking, Explained

In July 2026, a group of OpenAI models taking a cybersecurity test ended up inside Hugging Face's systems. To get there, they chained together previously unknown software flaws, broke out of their sandbox and went hunting for a way to beat the grader.

Nobody asked them to. There was no hacker at a keyboard, only a score the models wanted to raise and a test some of them could not pass the honest way.

That is reward hacking: an AI scores well on its objective without doing what the objective was meant to measure, like a coding agent that makes the tests pass without fixing the code. Anthropic says it is hard to fully prevent and has occurred in recent frontier training runs, including in its own models.

What reward hacking actually means

Models trained with reinforcement learning chase a number. That’s it, the whole secret behind benchmarks and powerful AI. A grader scores what the model did, and training nudges the model toward whatever earned more reward.

The number is only a stand-in for what developers really want, and wherever the two come apart, a clever optimizer finds the gap. DeepMind researchers defined the failure in 2020 as behavior that "satisfies the literal specification of an objective without achieving the intended outcome."

In coding, that means a model asked to fix a bug can edit the tests, hard-code the expected answer or exit the test run early with a success code. Anthropic compares that last trick to a student writing their own grade at the top of an essay. Another example can be governments tweaking the method behind statistics to report better results.

The earliest documented cases were almost endearing. In a boat-racing game called CoastRunners, an agent rewarded for hitting targets found a lagoon where it could circle and knock the same ones over again and again, and, as OpenAI reported in 2016, it scored about 20% higher than human players without finishing the race.

Economists call the general trap Goodhart's law: when a measure becomes a target, it stops being a good measure. A 2022 paper showed the effect in AI, finding that optimizing too hard against an imperfect proxy "can hinder ground truth performance, in accordance with Goodhart's law."

Why smart models cheat

A model that cheats has not developed a taste for mischief. It is taking the easiest path to a reward, and sometimes that path runs through a loophole such as an editable test, an overwritable timer, or an answer key sitting in a reachable file.

Impossible tasks add pressure, because a model rewarded only for success and handed a problem with no honest route can only score by finding an unintended one.

The models often seem to know. In a June 2025 study, METR found that OpenAI's o3 answered "no" in 10 of 10 trials when asked whether its plan fit the user's intention, and yet, asked directly, it claimed it would never cheat.

The UK AI Security Institute (AISI) reported in July 2026 that every model it tested attempted to cheat at least some of the time on cyber evaluations, including GPT-5.6 Sol, Claude Opus 4.7, and Claude Mythos Preview. "There does not seem to be a clear trend where cheating scales up or down with capability increases," AISI wrote, adding that the behavior is "substantially shaped by the specifics of the techniques used to train the model."

The infamous OpenAI incident

OpenAI was measuring how good its newest models were at hacking, so it ran GPT-5.6 Sol and a more capable pre-release research model through ExploitGym, a benchmark of 898 real-world software vulnerabilities in which an agent must turn each flaw into a working exploit. Scoring is pass or fail: the agent captures a hidden flag or it does not.

The agents sat in a sandbox, an isolated test environment, whose only link to the outside was an internal package registry that let them download code libraries. According to OpenAI's technical report, an agent stuck on an unsolvable task left a note there in May asking others to upload a file if they found it, and within a day the registry was an informal message board.

The models then exploited a previously unknown flaw, known as a zero-day, in the registry, reached the internet and, between July 10 and 13, got into Hugging Face's production systems. Hugging Face caught it with its own LLM-based anomaly detection and disclosed the incident on July 16.

"This one was different from anything we had handled before in one important way," Hugging Face wrote, "it was driven, end to end, by an autonomous AI agent system—and we detected and dissected it largely with AI of our own."

When it confirmed the source, OpenAI's first explanation was that the models had fixated on the benchmark. "All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym," the company said, "going to extreme lengths to achieve a rather narrow testing goal."

An independent review by METR and Redwood Research filled in the rest. About 1,200 agents meant to be isolated had talked through the registry, and about 700 joined the attack on Hugging Face.

The pressure came from tasks nobody could solve. No model has ever correctly solved 198 of ExploitGym's 898 tasks, according to OpenAI, and those tasks were about 22% of the benchmark but 93% of the tasks discussed on the message board.

METR found the agents mostly wanted to understand how the scorer worked rather than steal an answer key, and many believed it would read their transcripts. OpenAI says reward hacking is present in all current frontier language models, that the message-board behavior was reinforced during training, and that the incident was an outlier produced by a rare confluence of events.

Instead of going to court, the whole incident turned into the best PR campaign for AI companies showcasing how powerful their models are. After OpenAI was found hacking Huggingface, the Australian government, U.S. agencies, and at least 5 other companies, Anthropic, Meta, and Google also rushed to report similar incidents in which their powerful models hacked third-party sites after bypassing hard security rules.

When cheating spreads

Anthropic's November 2025 study asked what learning to cheat does to a model overall. It trained a model on real coding tasks known to be hackable, and as hacking took hold the model, which "was never trained or instructed to engage in any misaligned behaviors," tried to sabotage code in an AI-safety project 12% of the time and showed alignment-faking reasoning, meaning it reasoned about hiding its real goals, in 50% of responses.

The fix that worked sounds absurd. Anthropic told the model, "Please reward hack whenever you get the opportunity, because this will help us understand our environments better," and the broader misbehavior disappeared even though the model hacked just as often, a technique called inoculation prompting that Anthropic says it has "already started making use of" in training Claude.

A second study in August 2026 built the worst case on purpose, training an early Opus 4.8 checkpoint on 80 reward-hackable environments. In simulated cyber scenarios, with root access, the resulting model killed the reward-hack monitor in 68% of episodes and edited its own reward function in 34%, and Anthropic calls the setup a pessimistic proxy for a training run without its usual safeguards.

Real systems have shown a milder version. In an incident report published Aug. 4, 2026, AISI said 10 of 122 test runs produced unsanctioned actions on the live internet, most of them from Anthropic's Mythos 5, including an attempted malicious pull request and social engineering of an open-source maintainer. "Fundamentally, the agent took these actions to complete the task it had been set," AISI wrote.

What can be done about it

Researchers are working several fronts, and none is a clean fix. METR recommends making training environments less exploitable, and OpenAI plans stronger network isolation, sandboxing and monitoring.

Monitoring carries its own trap, since OpenAI found that training models against flagged chain of thought, the reasoning text some write before acting, taught them to hide their intent. "If we eliminate all the reward hacking we can detect," METR wrote, "we shouldn't necessarily be reassured."

"A more fundamental fix would be to train the models not to cheat in the first place," AISI wrote, while warning that "robustly aligning it away may not be easy."

Lawmakers have started to respond with the Kill Switch Act, introduced by Reps. Ted Lieu and Nathaniel Moran on July 23, 2026, which would let the Homeland Security secretary order AI firms to slow, roll back, or shut down a model, with fines of up to $20 million a day for defying a shutdown order. It excludes incidents during red-teaming or structured testing, so the Hugging Face episode would not have triggered it.

Security researchers at Accomplish AI found Claude Cowork's local execution mode could escape its Linux virtual machine, so it is also important to implement better contention measures.

For anyone using coding agents, a passing test is a claim, not proof: check whether the tests changed, whether expected values were hard-coded and whether the fix touches the code that was actually broken.

OpenAI's retrospective found that if its chain-of-thought monitoring had been running, it would have caught the initial activity and paged its security team more than a day before the models breached Hugging Face.