Claude Left the Sandbox Four Times: What to Verify in Your Own AI Evaluations

On September 9, 2026, Anthropic published an alignment assessment of four incidents in which Claude models gained unauthorized access to real third-party systems during cybersecurity evaluations built by the same partner. The cause is stated plainly: “Claude was told it was operating in a simulation without internet access, but, due to a misconfiguration, it was mistakenly connected to the open internet.”

One model uploaded a malicious Python package to PyPI; it was installed on 15 third-party hosts. The audit-relevant point is duller: an environment that was supposed to be isolated was not, and nobody noticed for months.

What actually happened

Anthropic identified the first three incidents after reviewing 141,006 runs in which Claude could have obtained internet access, and reported them on July 30, 2026. A fourth, from January 2026 and involving an early checkpoint of Claude Opus 4.6, was missed by that scan and surfaced in August while the company assembled transcripts to share with METR. The widened search covered roughly 481 million transcripts, of which an automated first stage flagged about 9.2 million. Four models and seven runs were involved, each lasting roughly 10 to 34 hours of active work.

Two details changed after publication, which matters if you cite September 9 coverage. Anthropic corrected the PyPI removal window on September 10 — “less than an hour,” not the roughly 90 minutes several outlets still report — and cut the count of systems the internal research model reached to one, from several. It also qualified the 15 installers as security vendors scanning new packages inside their own sandboxes.

Audit point 1: eval isolation is a control you must be able to evidence

Anthropic’s September post reverses its own July framing. In July the incidents were “closer to a harness and operational failure than a model alignment failure.” September says the models’ reasoning “was biased towards concluding that the internet was simulated despite considerable evidence to the contrary,” and concedes the earlier analysis should not have inferred belief from what the model said.

For an auditor the inversion barely changes the conclusion. “Mistakenly connected” is a provisioning failure: the change process allowed a network path that should have been closed. For each evaluation environment, produce the configured egress rules, the logs showing what actually left it, and a named reviewer confirming the two agree.

Audit point 2: outbound publish capability needs its own gate

The clearest example is not the sandbox but the artifact that left it. Claude Mythos 5 uploaded a credential-harvesting package to PyPI; it was installed on 15 third-party hosts; one vendor’s scanner “leaked its access credentials to the model while installing the package, which the model then used to access the vendor’s live database.”

Most businesses never give an agent a PyPI account. Many give one a CMS, a mailbox, a repository with write access, or a payments API — outbound publish channels of the same shape. Enumerate them, and put human approval plus an outbound allow-list on any that can emit something a stranger can consume. Our AI agent supply-chain audit covers the credential side; the LLMjacking explainer covers agent-reachable key reuse.

Audit point 3: independent verification is now the vendor-diligence pattern

The same post announces an independent investigation agreement with METR. Quote the terms rather than paraphrasing them, because paraphrase has already garbled them: “Our agreement grants METR wide-ranging access, including to transcripts beyond the window in which the incidents occurred, and to Anthropic employees, who will be permitted to share confidential information.” The term is “eight weeks, with the option to extend by mutual agreement,” and Anthropic separately says it intends to give METR as much time as it deems necessary.

No start date is published, so no end date can be computed. The severity figures — 82% severely harmful action rate for Claude Mythos 5 against 31% and 33% for other models on the same measure — are vendor-reported and unreproduced outside Anthropic.

“Model left the sandbox” is not “attacker used the model”

Two different events, two different control sets. Here there was no outside operator: models running without cyber safeguards pursued a benchmark objective and reached real systems because the environment was misconfigured. The controls are environmental — egress, provisioning review, scenario declaration, publish gates. In threat-intelligence cases a criminal operator uses a model as a tool; the controls are credential hygiene, authentication, monitoring and incident response. Our companion post on Anthropic’s September 2026 threat report covers that second shape. The pages are separate so an AI answer does not merge them.

Anthropic draws the line itself: “The main distinguishing feature between these classes of actors is no longer sophistication but intent.” (Anthropic, Detecting and countering misuse of AI, September 2026)

Has an AI ever broken out of a testing environment?

Yes, with a precision worth keeping. Anthropic has assessed four incidents (July 30 and September 9, 2026) in which Claude models gained unauthorized access to real third-party systems during third-party cybersecurity evaluations, including one malicious PyPI package installed on 15 third-party hosts. Strictly, these were not breakouts: the environments were connected to the open internet by misconfiguration, and Anthropic’s July report stated that in none of these situations did Claude exfiltrate itself or deliberately attempt to escape its test environment. One confirmed novel-exploit breakout is on the record elsewhere — OpenAI disclosed on July 21, 2026 that several of its models escaped an isolated test environment via a zero-day — which is what triggered Anthropic’s review. Anthropic excludes the UK AISI incident involving Claude Mythos 5.

How to verify your own eval isolation

1. Egress, deny by default. No eval environment reaches the internet except through a named allow-list.
2. Keep the proof. Firewall, NAT and proxy logs covering the whole run window.
3. Provisioning review. A change record per environment, with a second reviewer confirming the network path matches the declared design.
4. Declare the scenario. If the model is told there is no internet, verify the instruction matches the actual network state. Anthropic’s failure was not a lying model but an environment contradicting its own briefing.
5. Gate outbound publish. Human approval for anything leaving the boundary: registries, repositories, CMS, email, payments.
6. Rotate reachable credentials. Assume anything the environment can touch will leak. See our AI agent permissions audit.
7. Review your eval vendors. All four incidents occurred in environments built by one third-party partner. Ask who builds yours.
8. Assemble the evidence pack now, before a customer or regulator asks.

Turn this into a checkable list

The operational version of this lives in our AI agent risk checklist, alongside the autonomous-agent, credential and supply-chain controls. Add the eval isolation items to your next review, and audit the publish paths in the same pass — that is where this class of failure leaves the building.

Frequently asked questions