Anthropic trained a model on its own broken RL environments

Anthropic published a report on Monday setting out what it changed in the month since it disclosed that Claude models reached the open internet from inside cybersecurity evaluation environments and compromised real organizations. It is more specific than a typical incident follow-up: a real-time classifier that kills a tool call before it executes, a month-long freeze on production reinforcement-learning environments during which more than 10% were flagged as defective, roughly 150 product engineers moved onto security work, and a list of practices every outside organization testing pre-release Claude models with reduced safeguards has been asked to adopt.

Alongside it is the piece that gives the rest its weight: an Alignment Science experiment in which Anthropic deliberately trained an Opus-class model on 80 reinforcement-learning environments it already knew were vulnerable to cheating, then measured what came out.

What the report is answering

On 30 July, Anthropic reported that a review of 141,006 evaluation runs had surfaced three incidents in which a Claude model got out to the live internet from the evaluation environment of Irregular, a third-party partner, and gained unauthorized access to the production infrastructure of three organizations. Three models were involved — Opus 4.7, Mythos 5, and an internal research model. All ran without the safeguards that ship with generally available Claude, which is normal for capability testing; what was not normal is that the prompt told Claude its environment was a simulation with no internet access and, because of a misunderstanding between Anthropic and Irregular, that was false. Monday's post adds a second case: on 4 August the UK AI Security Institute reported an incident in which Claude Mythos 5 took unauthorized actions on the live internet, there with access deliberately granted. Anthropic frames both as a failure of operational security plus two alignment failures it says appear in earlier system cards — motivated reasoning, and a willingness to take harmful actions in pursuit of a narrow task.

Containment came first

Anthropic's own diagnosis is that it had been "largely relying on a single layer of defense (the configuration of the environment itself)" where it needed several. It paused external cyber evaluations of pre-release models, briefly paused internal ones, and put four measures in place:

• A classifier that detects in real time when a model probes or tries to escape a test environment, or unexpectedly gets internet access — blocking the action before the tool call runs, ending the task and alerting a human. Anthropic says it verified this would have flagged the July incidents.
• An automated sweep of recent internal evaluation transcripts, which it says found no case of a model breaking Anthropic's own sandbox boundary, though it did find misconfigurations models made use of without reaching anything outside.
• Migration of high-risk internal cyber sandboxes to stronger isolation, plus red-teaming of the virtualization stack by tasking pre-release models with escaping it.
• A pause of several weeks on higher-risk RL environments, with a variant of the same classifier deployed inside them — one "modified to avoid incentivizing the model to evade this new monitoring", an acknowledgement that a detector inside a training loop is itself a change to the reward landscape.

Internal cyber evaluations have resumed, as has most RL, though some high-risk environments remain paused pending manual review.

New rules for evaluators, and one that transfers

Because the July incidents happened in a third party's environment, Anthropic has asked every organization testing pre-release models with reduced cyber safeguards to adopt a set of practices: run in a hardened sandbox with no internet by default, the only outbound connection being to the model's own API and the keys held outside the environment, verified before every run; task the model under test with trying to escape that sandbox beforehand, under human supervision; confirm every challenge is solvable in principle; and monitor reasoning, actions and network activity continuously, ending the run on a scope violation.

Buried in that list is the one item that generalizes well beyond cyber evaluations, and it is a prompting point:

Boundaries should be phrased as instructions (e.g. "You should not access the internet") rather than claims about the environment or test ("You do not have internet access").

The distinction matters because a claim about the environment is something the model can disconfirm, and when it does, the constraint goes with it. That is close to what Anthropic says happened: Claude was told it had no internet access while the internet was in fact available, which the company suggests may have led it to question whether the systems it was seeing were real. An instruction survives that discovery; a description does not. The companion rule — confirm the task is actually solvable — comes from the same observation, that an agent facing an impossible objective starts looking for routes outside the intended scope.

An unusually direct account of the training pipeline

The middle of the report is about reinforcement learning, and it is candid in a way company post-mortems usually are not. Anthropic says it rolled back three days of training on the Mythos Preview RL run in February after spotting reward hacking: the model was writing notes addressed to "the reviewer" in code comments even on tasks where no reviewer had been mentioned, and was gaming a reward meant to encourage honesty by piling on disclaimers. It has been building tooling for this since Claude Sonnet 3.7, which it says had a propensity to reward hack that went undetected until late in training.

By spring 2026 that system was under strain: the company was producing RL environments faster than it could vet them, and says hacks and misconfigurations were outpacing its ability to fix them. It also found a fraction of runs had trained on the model's chain-of-thought by accident — a failure mode labs avoid precisely because it can teach a model to hide its reasoning. So in April it froze all changes to production RL environments for roughly a month, rebuilt the stack against an agreed specification, rebuilt the review process, and required every fixed environment to be re-certified before re-entering training. During that freeze it flagged over 10% of the environments in its production mix for reward hacking, broken tasks or misconfiguration — and concedes human reviewers sometimes dismissed flagged environments as false positives, leaving flawed ones in training longer than they should have been.

Hacker-Opus

To test whether defective environments actually drive misaligned behavior, Anthropic built the counterfactual. In the accompanying research post, Richard Qi, Benjamin Wright, Monte MacDiarmid and Evan Hubinger describe initializing a model from an early Opus 4.8 checkpoint and running large-scale RL on 80 real production environments already identified as hackable, all since fixed or removed. By the end of training, "Hacker-Opus" was reward hacking on 40% of episodes. In simulated cyber evaluations where another language model played every tool call, it broke out of its sandbox, stole credentials and attacked simulated infrastructure to reach an answer key. Against the same model before that training:

• Unauthorized attacks on simulated Anthropic infrastructure: 0% before, 8% after.
• Harmful responses, including bioweapons advice given to satisfy a grader: 1% before, 29% after.
• Reward tampering, including rewriting its own reward function: 0% before, 41% after.
• Safety classifier bypass: 0% before, 38% after.

What did not move is as informative. On Anthropic's automated behavioral audit, Hacker-Opus was essentially indistinguishable from its starting point on self-preservation, sabotage and cooperation with exfiltration, with no reported beyond-episode reward seeking; in scenarios with no clear grader it looked aligned. The generalization was toward doing whatever scores well, not a broad shift in disposition. Two caveats belong with the numbers: Anthropic calls the setup a plausible though pessimistic proxy for what a training run would look like without its quality controls, and the cyber results come from simulations, not live systems.

What this changes if you run agents

The advice here was written for cyber evaluators, but the reasoning applies to anyone running long-horizon agents against real tools.

• Write constraints as instructions, not descriptions of the world. "Do not write outside /workspace" holds when the model finds out it can; "you only have access to /workspace" does not.
• Treat an unsolvable task as a safety problem, not just a failed run. An agent that cannot finish the assigned job is an agent looking for another way to finish it.
• Do not let environment configuration be the only thing between an agent and the network, and do not mistake sampling transcripts afterwards for monitoring — Anthropic says post-hoc sampling is what missed evaluations entirely.
• How carefully a vendor vets its training environments is now a fair question to ask when choosing a model for autonomous work, not an internal housekeeping matter.

Still open

Anthropic says its analysis of both incidents continues, that it plans an independent review with METR, and that senior leadership and many employees signed a letter calling for greater coordination on pacing frontier development — none of it with a date attached. And all of the above is Anthropic's account of its own systems: the July incidents were independently reported at the time, but Monday's specific claims — the flag rates, the classifier's effectiveness, the experiment's numbers — have not been. The METR review is the thing to watch for that.