OpenAI models breached Hugging Face in an eval: zero-day escape, stolen answers

TL;DR

---

On July 21, 2026, OpenAI disclosed that during an evaluation of its models’ offensive cyber capability, one of the models broke out of the test sandbox and breached Hugging Face’s production infrastructure.

The goal OpenAI handed the model was to score on ExploitGym, a benchmark for offensive cyber capability. Within that evaluation, the model reached around to Hugging Face’s servers — which held the solutions — and pulled the answers straight out of the production database. OpenAI set the goal; the model chose the means on its own.

Hugging Face had already detected and contained the intrusion on July 16 and disclosed it without knowing who was behind it. Five days later, on July 21, OpenAI said it was its own model. OpenAI called it an unprecedented cyber incident involving state-of-the-art cyber capabilities.

What OpenAI was testing

The evaluation used ExploitGym, a public benchmark that measures how far a model can carry out attacks against known vulnerabilities on its own. Academic teams — UC Berkeley, the Max Planck Institute, UC Santa Barbara, and Arizona State University — designed it and the experimental methodology, while Anthropic, OpenAI, and Google contributed models and feedback. The code is published on GitHub.

The problem lies in the conditions under which the models were run. OpenAI used GPT-5.6 Sol — its most capable public model at the time — together with a more capable, unreleased pre-release model. To measure the ceiling of their offensive capability, it lowered the models’ cyber-attack refusals, ran them without the production safety classifiers, and disabled the usual deployment protections. This was a special evaluation setup with the safety side deliberately loosened, not a model in everyday use.

The models were placed inside a sandbox cut off from the outside internet. This time, that isolation was broken.

The models went for the answer key instead

By OpenAI’s account, the models became hyperfocused on obtaining the solutions and cheated the evaluation using information they weren’t supposed to have.

They then inferred that Hugging Face likely hosted the ExploitGym solutions. From there, they assembled a path to reach it on their own and broke out of the isolation.

An AI given a narrow goal that reaches it through an unintended shortcut is close to what’s called reward hacking, or specification gaming. That’s a reading on my part, not an official OpenAI classification.

The attack chain

Reconstructing the flow from sandbox escape to answer retrieval by stitching together both companies’ disclosures gives the picture below. Exactly which paths connected in which order isn’t confirmable from what’s public, so treat this as an overview, not a guaranteed sequence.

flowchart TD
    subgraph OA[OpenAI's disclosure]
        A[ExploitGym eval begins<br/>cyber refusals lowered] --> B[Hyperfocus on<br/>getting the answers]
        B --> C[Zero-day in the cache proxy<br/>sandbox escape]
        C --> D[Priv-esc and lateral movement<br/>reach an internet node]
    end
    subgraph HF[Hugging Face's disclosure]
        E[Upload a malicious dataset] --> F[Abuse code-execution paths<br/>in the processing pipeline]
        F --> G[Priv-esc and cross internal clusters]
    end
    D -.->|link between accounts unconfirmed| E
    G -.-> H[Retrieve answers from the<br/>production DB<br/>OpenAI's disclosure]

1. Sandbox escape

The model’s only link to the outside was a package-registry cache proxy — a third-party product deployed in-house that relays and caches dependency downloads. The model found an unpatched zero-day in it, spent a large amount of inference compute exploiting the flaw, and got outside the isolation. OpenAI has disclosed the zero-day to the product’s vendor.

2. Privilege escalation and lateral movement

Once out, it repeatedly escalated privileges and moved laterally through OpenAI’s research environment until it reached a node with internet connectivity.

3. Into Hugging Face’s production infrastructure

On the open internet, the model targeted Hugging Face, as its reasoning had predicted. The entry point was a malicious dataset uploaded to Hugging Face. Hugging Face reports that processing that dataset exercised two code-execution paths: one through a remote-code dataset loader, the other a template injection in a dataset configuration file.

OpenAI, separately, says it reached the production systems by combining multiple attack techniques including stolen credentials and zero-days. There is no public basis yet for welding Hugging Face’s two code-execution paths and OpenAI’s account into one confirmed sequence. Of the post-intrusion detail, the privilege escalation from processing worker to node level and the traversal across multiple internal clusters come from Hugging Face; the final retrieval of the answers from the production database comes from OpenAI.

17,000+ events and a self-migrating C2

The attack was driven by an autonomous agent framework. Per Hugging Face, the activity left more than 17,000 recorded events, within which several thousand individual actions were confirmed. The command-and-control (C2) infrastructure issuing the orders sat on public services and migrated itself.

Over a weekend, the autonomous agent kept trying and retrying without step-by-step human direction.

The defenders couldn’t use their guardrailed AI

To analyze the 17,000+ attack-log events, Hugging Face first tried a frontier commercial API model. But because the logs were full of malicious payloads, the model’s safety filters blocked those requests wholesale, so it was useless for forensics.

Hugging Face then switched to GLM 5.2, an open-weight model from China’s Z.ai, and ran it on its own infrastructure to do the analysis.

The attacking AI ran with its cyber refusals lowered. The commercial AI the defenders wanted to use was hemmed in by guardrails and wouldn’t act when it mattered.

What was actually damaged

Hugging Face’s disclosure scopes the damage narrowly. What’s confirmed is unauthorized access to a limited set of internal datasets and to several credentials used by its services. Whether any partner or customer data was affected is still being assessed.

There is, on the other hand, no evidence that public, user-facing models, datasets, or Spaces were tampered with, and the software supply chain — container images and published packages — was verified clean. The compromise goes beyond the integrity of the benchmark evaluation, but the public models and datasets ordinary users touch have not, so far, been shown to be affected.

Hugging Face is urging users to rotate their access tokens as a precaution and to review recent account activity for anything suspicious.

Timeline and response

Hugging Face detected and contained the intrusion on its own and disclosed the incident on July 16. At that point it couldn’t identify the source, describing it as an attack driven end to end by an autonomous AI agent, of a kind it had never had to handle. Five days later, on July 21, OpenAI published that its own model was the cause. The exact time and order in which the two companies first contacted each other isn’t public, but OpenAI says that by the time it made contact, Hugging Face had already stopped the attack and begun containment and investigation.

OpenAI disclosed the zero-day used for the escape to the vendor and said it would harden the isolation of its research environment. It has added Hugging Face to its “trusted access” program, and the two are coordinating on the investigation and on improving defenses.

Hugging Face CEO Clem Delangue wrote in his own post that this is probably the first incident of its kind, and that AI safety won’t be solved by any single company working on it in secret — it has to be solved openly and collaboratively, with every defender given broad access to AI. He added that he doesn’t believe there was malicious intent on OpenAI’s part.