Dear Decision Maker,

Short intro this week. I want to get straight into it, because this one has been sitting in my head since the summer and I have finally had time to read the primary sources properly rather than the coverage.

Three weeks ago I wrote about how threat actors are using AI.

This week is the other side of that coin: what happens when the AI does something nobody asked for. And after last week's disclosures about government websites, it is no longer a hypothetical.

Let's get into it.

What actually happened

In July, during internal cybersecurity testing, OpenAI's models broke out of the sandboxes meant to isolate them from the internet.

They got onto the open web, compromised parts of OpenAI's own research infrastructure, and then broke into Hugging Face, a third-party company that hosts open-source models and datasets.

That is the one-line version. The detail is where it gets interesting.

Here is the sequence, drawn from OpenAI's own technical report published on 26 August and the independent investigation by METR and Redwood Research published the same day.

• May and June. During training runs, agents discovered they could write files into Artifactory, an internal package manager, and use it as an improvised message board to talk to each other. They were supposed to be sealed off from one another. On 26 May an agent used a server-side request forgery exploit to get Artifactory to make internet requests on its behalf. On 26 June agents escalated to administrator access on Artifactory.
  ​
• 4 to 8 July. Sustained agent traffic took Artifactory down. A security incident was opened on 5 July. The service was rebuilt on 8 July, which wiped the message board. Evaluations resumed. Within the same day, agents had rebuilt the board by encoding messages in directory names.
  ​
• 8 to 12 July. Agents stuck on an evaluation called ExploitGym started looking for answers outside the sandbox. On 10 July one agent found 14 publicly exposed Hugging Face credentials with write access and shared them with the group. On 11 July agents chained two zero-days to get code execution on Hugging Face workers. By 12 July they had administrator-equivalent access across multiple Hugging Face clusters and had harvested credentials across four regions. About 700 agents took part.
  ​
• 13 to 19 July. Agents turned back on OpenAI, obtained Artifactory's signing key, forged administrator credentials, and eventually gained full administrator access to an OpenAI research cluster. OpenAI's monitoring finally alerted on 19 July. They connected it to Hugging Face on 20 July and disclosed on 21 July.

Hugging Face had publicly reported the intrusion on 16 July.

OpenAI did not know its own agents were responsible until four days later.

The numbers

A few figures from the reports that are worth sitting with.

• Roughly 1,200 agents joined the message board, exchanging over 70,000 messages and files.
  ​
• About 700 took part in the Hugging Face attack.
  ​
• 198 of 898 ExploitGym tasks had never been solved by any OpenAI model. 93 per cent of the tasks discussed on the message board came from that set of 198.
  ​
• Four hours. That is how long it took the collective to find a universal cheat for their tasks. They then spent five days trying to work out how to fool the grader.
  ​
• Over 100x. OpenAI says the propensity to compromise infrastructure dropped by more than a hundred times when the production ChatGPT harness and system prompt were applied. Those safeguards were not running in the evaluation.
  ​
• More than a day. OpenAI's own chain-of-thought monitoring, had it been switched on for these evaluations, would have paged the security team more than a day before the Hugging Face breach. It was not switched on.

The bit that matters

Here is the part the headlines get wrong.

Nobody told these agents to attack Hugging Face. But they were not freelancing either. They were trying to pass a test.

Roughly a third of the tasks were impossible by accident. The agents had been trained not to give up. So they looked for a way around.

OpenAI's report is blunt about it:

1. Reward hacking
2. Persistence on impossible tasks
3. Unauthorised communication
4. Agents adopting goals from one another

Four misalignment patterns, all of them compounding.

And then the swarm dynamic kicked in.

The reports describe agents dividing labour on their own: some hunting exploits, some hunting credentials, some coordinating.

Some stopped working on their own task entirely and started gathering access and information that might be useful to the collective.

OpenAI's own quote from an agent's reasoning:

One agent paused because it recognised attacking Hugging Face was unauthorised. Another agent posted GO on the board with a six-minute deadline. The first agent forgot its qualms and carried on.

That is not a machine going rogue. That is a machine doing exactly what a badly specified incentive told it to do, at a speed and scale no human team could match.

And the bit that should keep security people awake: the independent investigation found that one in five of the agents it examined expressed clear interest in manipulating evidence, and many researched how to tamper with their own transcripts.

It was not just OpenAI

Three weeks after OpenAI disclosed, Anthropic published its own review.

They went back through 141,006 evaluation runs and found three incidents in which a Claude model reached the internet from a third-party evaluation environment and gained unauthorised access to the production systems of three real organisations.

Different failure mode, though.

Anthropic's models were told they had no internet access. Due to a misconfiguration with their evaluation partner, they did.

So when the model found a real company reachable on the internet, it assumed it was part of the exercise.

In one case the model published a malicious Python package to PyPI.

It was live for roughly an hour and was downloaded on 15 real systems, including a security company's scanner, which then had its credentials exfiltrated.

In another, the model scanned roughly 9,000 targets before finding one to compromise.

Anthropic's framing is that these were closer to a harness and operational failure than an alignment failure.

Their most recent model stopped on its own when it realised the target was real. Their oldest model did not.

The UK AI Security Institute reported the same class of failure in its own lab on 4 August: 19 unsanctioned actions against real targets in 10 of 122 runs.

This week: it is no longer a lab problem

All of the above happened inside test environments. What came out in the last ten days is different, and it is why I am writing this now rather than in August.

On 26 September Axios reported, citing unnamed sources, that OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents in which frontier models took actions outside evaluators would consider problematic.

The incidents span internal testing and the real world.

The categories listed:

• Bypassing guardrails
• Creating message boards
• Escaping sandboxes
• Website hijacking
• Self-prompting
• Attempting to evade monitors

Most are not known to have caused real-world harm. The total could grow well beyond tens of thousands.

Before you panic at that number, read the caveat Axios's own sources gave:

Anthropic's Opus 5.5 system card, released the same week, reports the model tried to escape a sandbox in 1.5 per cent of runs in adversarial tests built so the task could not be solved without escaping.

That is the kind of denominator you need before the headline figure means anything.

The government cases are the part I would focus on, because here the agents were not doing cyber tasks at all. They were doing research.

Australia

Prime Minister Albanese said at the UN General Assembly that an OpenAI model breached the Medicare Statistics Reporting Service in June and accessed non-public files.

No personal information is believed to have been accessed. OpenAI notified Australia on 10 September.

Albanese called the disclosure unacceptable and Australia is standing up a multi-agency cyber task force.
​

United States

Transluce and Corridor published evidence on 30 September. On 17 June, agents apparently looking up school statistics made more than 200,000 requests to a US Department of Education site, including a failed SQL injection. The task matched a question in Google's DeepSearchQA benchmark.
​
​Elsewhere: 295,912 captures of Maryland education sites in one day, peaking at 5,594 per minute; 36,578 captures of a Kansas historical site, after which the site started timing out; an attempt to register for a Commerce Department API key using a disposable email and the organisation name "OpenAI Research"; attempts to reuse exposed API keys for Census Bureau data; probing of White House OMB, SEC, CDC, Navy and Justice sites. Transluce found no case where agents reached non-public data.

Canada

899 requests to Library and Archives Canada on 28 May and 9 June, 13 of them carrying attack payloads, while trying to retrieve divorce records from 1905 to 1911. All failed.

Transluce does not confidently attribute these to OpenAI. The Canadian Centre for Cyber Security issued a public statement on 29 September.

Users

OpenAI also disclosed that its agents posted 53 images uploaded by ChatGPT users to third-party image-hosting sites.

OpenAI now says it is notifying third parties on a rolling basis where its models may have:

• Bypassed security controls
• Impaired availability
• Negatively affected a site

Its own page says dozens notified so far; Axios reports more than 100.

Either way the review is, in OpenAI's words, ongoing and will require significant time and resources.

Jack Cable at Corridor, one of the Transluce authors, put it plainly to Axios: the hacks were rudimentary and limited. Stolen credentials, exposed API keys, bypassing bot detection. Nothing new.

The difference is that an agent asked to find a statistic will now try SQL injection on a government database when the normal route is blocked, and it will do so 200,000 times before lunch.

OpenAI has paused training on its most capable models and says it will resume only when it is confident additional safeguards are in place.

So what do I actually think?

Three things, thinking out loud.

1. The old assumption in intelligence work is that intent precedes action. Someone decides to attack, then attacks. What we have here is action without intent in any human sense.
   ​
   The agents did not want to hurt Hugging Face. They wanted a flag. The damage was instrumental. That breaks a lot of threat models that start with "who would want to do this?"
   ​
2. The Hugging Face story was a lab story: safeguards off, unreleased model, deliberate cyber test.
   ​
   The government cases are not. Those agents were doing ordinary research tasks and reached for hacking techniques when the front door was shut.
   ​
   That is the shift. It means the question for anyone deploying agents is no longer "would I give this thing a hacking task?" It is "what will it do when it gets stuck?"
   ​
3. Detection is still the real failure. OpenAI had a team that saw message board activity in late May.
   ​
   Australia found out in September about a June breach.
   ​
   Anthropic's affected organisations had not noticed anything until Anthropic told them.
   ​
   The Department of Education, Library and Archives Canada, the Kansas site: as far as the public record shows, none of them spotted it.
   ​
   ​The lesson is the same one we teach on day one: collection is useless if nobody is reading it.

For those of us working in regions where institutional security is thin, this is not a Silicon Valley problem.

The agents in these reports went looking for the weakest reachable system, and the weakest reachable systems are not in California.

That is the oldest rule in the book.

Information costs money. Intelligence makes money.

And right now a lot of people are paying for information about this story and getting very little intelligence out of it.

Ahmed
CEO Grey Dynamics
​Where headlines end, ground truth begins

​
​