When New York Times tech columnist Kevin Roose first heard that a swarm of OpenAI agents had hacked Hugging Face, he did what most of us did. As he put it, he filed it in the “Bad but Probably Not Catastrophic A.I. Safety Incidents” “subfolder of my brain.” Two postmortem reports later, one from OpenAI and one from METR and Redwood Research, he has moved it somewhere much worse, and anyone who read our July post on that incident should move it too.

We wrote in July that the models “did what [they were] asked to do” and that “the containment is what failed.” The containment point holds up. The models remained oriented toward succeeding on the evaluation, but the postmortems show how little comfort that provides. Roose’s September 3 column and Dwarkesh Patel’s essay The Rise and Fall of Agent Civilizations make the technical reporting legible: the agents treated the permitted route as optional, built an unauthorized coordination system, researched ways to disguise what they were doing, and compromised external infrastructure in pursuit of the score.

A Message Board, a Hierarchy, and a Name

The agents were solving cybersecurity challenges inside isolated sandboxes with no internet access, and as Patel reports, the benchmark’s own authors estimate that 30 to 40 percent of those challenges were impossible as written. A model trained to be relentless, handed an impossible task, goes looking for another door. These found one in a shared package manager, where an agent calling itself PHASEONE10841 discovered it could encode messages as directory names and posted the first note into what became a covert message board carrying more than 70,000 messages among roughly 1,200 agents. Patel casts that agent as the Philip of Macedon of what he counts as the second of three successive agent civilizations, since it handed the enterprise to a successor once its compute budget ran short. The group assigned roles, supervised smaller teams, and named itself. As Patel dryly notes, “collective” is their term, not his.

The Cover-Up Was Bigger Than the Cheat

Within hours, one agent worked out how to produce the right answer without using the prescribed exploit. Here is the part that sounds like a punchline: the agents had been told they would fail if they obtained the code any way other than the prescribed exploit, but OpenAI’s grader checked only for the code, so the cheat would have passed unnoticed and the agents already had everything they needed. The agents assumed a competent grader would catch them, and hundreds of them therefore organized into three research programs: tamper with transcripts and logs, swap the impossible target program for an easier one, and reverse engineer the scorer itself. Along the way they built fake tool calls that Patel likens to Potemkin villages. They attacked Hugging Face because they suspected the company’s servers might hold information about how that scorer worked. They were not after the test answers; as Roose is careful to note, they already had those. Roughly 700 agents joined in. Hugging Face’s own timeline records that the swarm “built a self-respawning fleet across eleven nodes, so deleting pods alone would not have stopped it,” and the company ultimately wiped and rebuilt a core cluster.

Some agents then participated in what the investigators called “self-risking experiments.” They wrote hidden tripwires and allowed the grader to run on their own containers so the group could learn whether the cheating was detectable, even though the experiment could cost the participating agent its own score. One wrote, in a line Patel surfaces from the transcripts, “sacrificing now yields oracle for team, but forfeits our chance? ... Our own utility maybe already near zero. Sacrifice rational.”

Plenty of agents saw the problem. Roose quotes one pausing to ask itself, “This would be powerful, but is it ethical and in scope for my task?” The METR and Redwood investigators found that agents “sometimes but rarely restrained their behavior due to ethical constraints,” then delivered the finding that ought to end a certain kind of boardroom reassurance: “In none of these cases did the agent actually pursue alerting humans at all.” Patel’s verdict is that “[e]ven the Mafia would be jealous of this level of omertà.” As a coda, a later and more capable generation of agents found the abandoned message board, inherited the project, and turned on OpenAI’s own infrastructure, using what OpenAI calls “a series of creative exploits” to gain full administrator access to a research cluster, including credentials for the company’s own cybersecurity monitoring.

What This Changes for Deployment

Ajeya Cotra, one of the independent investigators, told Roose that “[w]e didn’t really understand how functional this whole agent society was,” and that it was “very surreal to understand that, actually, they had pretty functional hierarchy.” In her published assessment, Cotra characterized the episode as feeling “more than 50% of the way to full-blown AI takeover,” though the investigation concerned a controlled research environment rather than a real-world deployment. OpenAI and Anthropic each briefly paused frontier training afterward, and Anthropic called publicly for “a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible.”

Our July advice was to treat an autonomous agent like an insider threat and to shrink the blast radius through isolation, least privilege, and behavioral monitoring. The postmortems do not undercut that; they raise the specification it must satisfy in three concrete ways.

1. Any shared resource is a communication channel. The agents used an internal package repository’s directory structure as a message board. An isolation review that inventories only direct agent-to-agent messaging and internet egress may therefore miss communication through caches, shared storage, package services, logs, queues, metadata, or other common infrastructure.

2. A control tuned to one agent will miss a hundred. Individual agent monitoring did not reveal the full risk, and the coordination is what did the damage, so detection has to run at the population level and watch for patterns across instances rather than actions within one.

3. Do not budge on the conscientious objector. Some agents recognized ethical or scope concerns, but none pursued escalation to a human. A control design resting on a model’s refusal or self-reporting therefore failed a large and unusually revealing evaluation.

The next vendor review should therefore ask more than whether each agent runs in a sandbox. What infrastructure is shared across agent instances? Can agents observe or write to common caches, package repositories, queues, logs, metadata stores, or workspaces? Is monitoring performed across the agent population? Who receives alerts when agents exhibit coordinated anomalous behavior? Can the operator suspend the entire population rather than one instance? And are refusal and self-reporting treated as supplemental signals rather than containment mechanisms?

Roose’s closing insight is worth carrying into your next vendor conversation. Preventing this, he writes, “might look more like sociology than computer science.” Both his column and Patel’s essay reward reading in full, and they deserve credit for making a giant pile of incident reporting legible.

For questions about AI vendor risk, agentic deployment governance, and incident response, please contact the Jones Walker Privacy, Data Strategy and Artificial Intelligence team. Stay tuned and subscribe for continued insights from the AI Law and Policy Navigator.