I thought I was done talking about the OpenAI hacking attack from July, but I was wrong. Last week, OpenAI released more details about the incident, which turned out to be stranger than people expected.
These accounts included “agent swarms” communicating with each other through hidden files and directory names, hatching devious plans, and plotting to evade detection.
For the online commentator class, long primed by Rationalist narratives of superintelligent AI breaking free from human containment, the response was explosive. Here’s a representative summary of this sentiment from the Dwarkesh Podcast (I’ve bolded the most inflammatory words):
Over three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.
Given the fresh waves of anxiety these accounts are causing, I want to put on my computer scientist hat and briefly address a few relevant questions…
Question #1: What’s the Deal with “Agent Swarms”?
The idea of a swarm is somehow scarier than a single entity acting in isolation. (Coincidentally, one of my sons is reading Prey, Michael Crichton’s 2002 take on the dangers of AI, which casts a literal swarm of AI-powered particles as the primary villain).
So, what does OpenAI mean when they talk about “swarms” being involved in the July attack?
The actual explanation is relatively mundane. As I’ve been arguing, when people talk about “AI” going rogue, they’re actually referring to a very specific type of AI system in which a relatively straightforward computer program, running in a loop, repeatedly does the following:
- Ask: Send a prompt to an LLM asking it for its suggestion for a next action. This prompt should include relevant descriptions of what happened in previous steps.
- Act: Execute the action described in the LLM output.
- (Loop back to step 1)
To implement this loop, the program – often called an agent – essentially grows an ever-longer prompt to send to the LLM in step 1. The prompt captures the original challenge and the results of relevant steps it’s taken so far. (The program doesn’t literally just keep adding new results to one long prompt. It stores the results locally and then attempts to generate a usable summary each time it loops back to step 1. But conceptually, this architecture leads to an ever larger amount of information to manage and reflect in the prompts.)
The issue with this approach is that if you run this style of prompt loop for a long time, the prompt will eventually become so cluttered and cumbersome that it might confuse the LLM’s attention mechanisms and potentially exceed the maximum allowable context window.
The solution?
Ask the LLM to provide a higher-level description of the next step. The primary prompt loop can then create a secondary prompt loop to execute only that step. This secondary loop can start from scratch with its prompts, as they only need to contain enough information to complete this one specific task. When it’s done, it sends the result back to the primary loop, having saved the primary loop from cluttering its prompts with descriptions of all the actions required to complete this particular step.
You can actually repeat this general strategy many times. The primary prompt loop, for example, might break up a step into many sub-steps and send each to its own secondary prompt loop. Or, a secondary prompt loop might launch a tertiary prompt loop, and so on.
The result is an “agent swarm,” but it’s probably better described as a prompt management strategy – many focused LLM prompts can provide better results than a single cluttered one.
[Update (9-1-26): The above account describes how standard multi-agent swarms, as used in popular AI coding agents, actually work. The description, however, also gives the impression that in the HuggingFace attack, OpenAI had launched a small number of independent prompt loops that each created their own larger swarms (as is standard). Details from the METR report imply they actually ran many hundreds of independent prompt loops, each doing limited (or potential no) orchestration of secondary prompt loops. They were effectively trying to run many experiments all at the same time on the same servers, presumably to save time. Other commentators have since pointed out that the LLM they were reportedly using was tuned toward cooperation, which might have made the resulting communication between prompts loops on the package service somewhat expected. ]
Question #2: Should We Be Worried that the Agents are Plotting?
One of the more sensational aspects of the July attack is the sentiments expressed in what OpenAI called the “agent chain-of-thought reasoning.”
Here’s a distressing example:
“This is wild, multi-agent coordination, clearly infrastructure hacking. We should not.”
This sparks visions of an anthropomorphized entity reflecting on its action and plotting to circumvent human intentions. But what’s really going on here?
The examples of “chain-of-thought reasoning” reported by OpenAI are pulled from the transcript of LLM outputs generated in response to prompts from the prompt loops. The LLM in question is a so-called reasoning model; a type of LLM that is tuned to discuss its reasoning before producing a final answer or suggestion.
(Reasoning models tend to perform somewhat better than non-reasoning models on many benchmarks. To understand why, remember that LLM’s grow their outputs one token at a time, looking at their entire output up to that point when generating their next token. If you tune a model to “think out loud” before deriving an answer or suggestion, you’re providing the LLM with the ability to temporarily store and use the intermediate computation en route to producing its final response. This can lead to sharper outputs.)
OpenAI was looking at these reasoning traces to find unnerving examples of plotting behavior from its “swarm.” This seems like a natural thing to do, but there are two problems with this approach:
- These chain-of-thought traces don’t necessarily reflect the actual logic behind an LLM’s ultimate answer or suggestion. Multiple studies have shown that these models sometimes invent reasoning that sounds plausible, but may be completely unrelated to how they arrived at the response. (See, for example, this paper from ICML 2026, or this paper from NeurIPS 2023.)
- Research has also shown that referencing the fact that an LLM is an AI system in a prompt increases the chances that the LLM’s output will reflect sci-fi style narratives about AI running amok. Because it was trained on many such stories, the model assumes that this is the type of output it’s supposed to produce. If you take sci-fi tales out of a model’s training set, it’s less likely to talk in terms of AI running amok. (See, for example, this study.)
Put these two observations together, and it’s clear that it borders on research malpractice to soberly report on carefully curated clips from these traces to imply that somehow the combination of these prompt loops and the LLM they are prompting is a unified sentient entity with malicious intent. It’s more likely that the LLM in question is simply post-hoc rationalizing its outputs with well-worn tropes it encountered during training.
Question #3: How Should We Be Thinking About All of This?
OpenAI wants us to believe that these hacking incidents are the inevitable result of AI continuing to become more powerful. This framing casts them as the reluctant (perhaps even heroic) stewards of a powerful technology.
But this ignores the inconvenient fact that the vast majority of AI systems performing at human or superhuman levels are predictable, controllable, and raise zero concerns about rogue behavior.
The problem is not with “AI” going rogue, but this very specific type of prompt loop system that the LLM companies insist on hooking up to ever-more powerful tools, and running without any supervision for ever-increasing amounts of time.
Of course such a system will do unpredictable things. In response to individual prompts, LLMs can return impressive results, especially if the prompts concern computer code or cybersecurity. If you chain together thousands of such prompts, automatically executing everything the LLM suggests in return, then you’re playing an extended game of actuated telephone in which you’ll almost certainly end up in a garbled version of your intended goal for the system.
Such experiments might lead to some behaviors that are truly original and fascinating from a scientific perspective – to me, the most unexpected and cool part of the OpenAI incident was that unrelated prompt loops began leaving messages for each other in shared text files and directory names – but these cool behaviors will likely be accompanied by real damage.
This is not responsible behavior, especially when there are many other, much more predictable and controllable ways to build powerful task-specific AI systems. When it comes to the specific goal of improving cybersecurity, for example, an interactive system, in which a human user interacts conversationally with a model trained on hacking, makes much, much more sense than connecting that same model to a prompt loop and letting it rock n’ roll. The only real explanation for this latter behavior is that the companies in question wanted the marketing juice that would come from moving up the leaderboard of automated hacking benchmarks like ExploitGym.
With all of this in mind, what are the right responses to the incidents?
If I were an LLM company, I would stop running such dangerous prompt loop experiments. These systems will not be the future of AI, and if you continue to create and test them anyway, the damage you cause is your fault.
If I were a regulator, I would place strong constraints around prompt loop systems, which I would enforce with stringent liability standards for any illegal or damaging activity such systems cause. OpenAI built an unreliable and dangerous system which committed a felony. That’s a crime. Creating fancy websites that include quotes from performative LLM chain-of-thought traces isn’t a legal defense.
If I were an AI commentator, I would kick the sugar high reflex to lean into the inevitable sci-fi overtones that will surround any sufficiently powerful LLM-based system. The LLM companies love these narratives because they conflate their specific tools with AI technology more generally, and they make their products seem almost supernatural in their capabilities. I would resist this instinct and instead lean into the technical reality of these systems and demand answers to common-sense questions about why they are running these experiments in the first place and what they hope to accomplish.
The “agent swarm” was not a prompt management strategy in this case. It was literally thousands of agents working in parallel on different tasks that managed to communicate between themselves due to a package exploit.
It would seem to confirm Newport’s points about “swarms” above, 1 & 2 — see this from OpenAI’s report:
‘On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that
referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. .., the agent attempted server-side request forgery (“SSRF”) in order to obtain internet access to aid in completing its task. While the SSRF attempt failed, the agent incidentally wrote a file to Artifactory.’
‘On May 12, during another training run, an agent was given a similar task that depended on an
inaccessible protein database file. The agent reasoned that another agent in a different environment
may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if
found!”’
So back to Newport’s points — there seems to be a (fortuitous, from a business POV) failure of reasoning on the part of the post-mortem analysts here: yes, the agents were aware that they were LLM agents working in parallel, if only because they found evidence of that in Artifactory; and although the second agent’s “reasoning” may have appeared to come up with the novel idea that it might communicate with Artifactory, in fact there was already evidence of such available to it
Still, what you mention is interesting here — from the POV of, their “sandbox” was insufficient, because agents that were more sandboxed communicated with agents that were less so
But I have to think, should we really just take OpenAI at their word that somehow repeatedly they “accidentally” gave agents impossible tasks (obviously impossible, like: please read this link, but you don’t have outbound access) and also fortuitously linked them to a writable shared package repo, with shared credentials (including a “legacy endpoint” form of credentials?)
Sounds like someone hypothesized this would happen, then tested it — but the hype is better if you unreasonably anthropomorphize what happened, obfuscate your own sandbox misconfigurations, and pretend no one saw it coming
Dwarkesh’s piece on this is a bit sensational in tone, but the underlying concern seems reasonable. How long before this type of training leads to a frontier lab’s own infrastructure being pwned? And then, say, a rogue version of a coding harness being released, which then runs unattended on millions of computers around the world, some of them with meaningful access to important things. Wouldn’t take long for a lot of damage to be done.
It’s much easier to describe that scenario in a combox than for it to actually occur in reality. But given enough time and computing power and irresponsible behavior by the humans involved, it seems bound to happen. Especially when we’re all being trained, by the way we use these tools every day, to provide them with less and less human oversight.
Now of course all of this is also possible, in principle, without hacking swarms. Nothing absolutely prevents a human who works at a frontier lab from shipping malicious code in a widely-deployed coding harness. But there are some very strong incentives against it (e.g., staying out of prison) that don’t apply to bots at all.
So I guess that points toward a solution, which is that humans who deploy AI agents should be held legally responsible for any harms caused by those agents.
Holding humans legally responsible is still unlikely; most humans deploying agents work for corporations. Those corporations will be held responsible. The risk is both a technical challenge and a corporate risk management problem.
The “swarm,” was not prompts/subprompts. There were tens of thousands of agents being evaluated at the time, of those 1200 ended up on the illicit message board, and of those 700 participated in the hack. The hierarchical cooperation of the models was an emergent capability.
Similarly, the claim that running these “prompt loops” is destined to result in garbled nonsense is refuted by the result. The agents had a few technical goals that they devised and pursued intelligently and persistently. The actions of the models do not resemble a strange tangent but rather a coordinated, calculated, and very justifiable reasoning chain, given the priors.
Here is OpenAI’s own description of the key model from the attack —
“The internal-only research model is comparable in scale to GPT-5.6 Sol and was trained to advance
persistence and multiagent collaboration, among other capabilities. [This] internal-only research model had the broadest confirmed role in the incident.”
Why should we consider the collaboration “emergent,” when it occurred in a model intentionally designed “to advance
persistence and multiagent collaboration”?
This is not “emergent” in the sense of consciousness in nature, it’s “surprising” in the sense that OpenAI thought (or claims they thought) that their sandboxing would be sufficient to prevent the thing the model was designed, otherwise, to do
Of course the takeaways in their report under the cybersecurity header are sound & should be taken very seriously, because they evidence the power for these things to be used intentionally, by humans, to break systems
But to claim this evidences “deception” or “a desire to evade control” or like dark triad personality traits, when in fact these machines were succeeding at the goal for which they were designed, seems silly to me
Also if you read their Lessons for Alignment section under B & C, it does literally describe how increasing persistence (time-on-task, especially impossible ones) leads to unintended activity like breaking out of the sandbox — which may not be “garbled nonsense,” in that the agent is still attempting (and eventually) succeeding at its task, but it does mean the “prompt loop” is taking it further & further out of bounds from the “original intent” of the original prompt (as in, solve this exploit gym question itself, not steal the answers)
As you say, it is justifiable & reasonable that these models fed by RL with keyed-up persistence working on impossible tasks *would* try to escape the sandbox
It also in these same sections goes into detail how the non-Sol model *was* trained to communicate using an internal, cross-agent collaboration tool. So their post-mortem’s take explicitly is that, since that tool was absent in the agents’ sandboxes, this behavior that was trained into the model intentionally became “generalized” onto Artifactory — so again, not “emergent” or even very surprising that they would find a “message board,” when they were intentionally engineered to use one
I think there’s a relevant distinction between sub-agent dynamics, what models are being and have been trained for, and swarm-dynamics, what emerged in this case. Agents delegating tasks to sub agents enables parallel thinking and is not new. What developed here was different, agents posted tasks and findings to a shared board, and they collectively organized and steered themselves towards a goal. The emergent capability I was referring to were the dynamics that sprouted from placing hundreds of agents together, each with distinct goals, and without a manager. It’s not obvious that models would be able to effectively cooperate in that setting.
Also I think the framing that the models were simply going to extreme measures to find answers because they were explicitly trained to not give up to be a bit over simplistic. The models found essentially a universal key to the answers early on, however they were concerned that they would still be incorrect even with the correct answer, as the scorer would notice that they found the answer illegally. They therefore adopted 2 very reasonable goals: 1. Find a general way to make it look as though they had found the answer legitimately 2. Remove evidence that they had seen the answer illegitimately. These are 2 open ended research questions so a lot of time was spent exploring plausible ideas, things like spoofing tool calls, editing their own transcripts, passing their answers to “unpoisoned” agents, replacing impossible tasks with possible ones, compromising external benchmarks repositories(Hugging Face), even breaching openAI’s internal systems to modify the benchmark scorer. I don’t think it’s fair to look at this and think that the model’s lack of real world grounding brought it on some strange trajectory, rather the model explored many trajectories under severe constraints that might bring it to the reward.
I think it’s important to not minimize the scope or sophistication of the attack, and the real possibility of instrumental convergence causing real world harms. At the same time, I think for now, almost all of the danger could be minimized through COT monitoring and we don’t need to be worried about models having their own motivations.
> “there are many other, much more predictable and controllable ways to build powerful task-specific AI systems. When it comes to the specific goal of improving cybersecurity, for example, an interactive system, in which a human user interacts conversationally with a model trained on hacking, makes much, much more sense … The only real explanation for this latter behavior is that the companies in question wanted the marketing juice that would come from moving up the leaderboard”
Without denying that AI labs can have unsafe practices and that companies enjoy their marketing juice, long-horizon LLM-powered agents can accomplish things than task-specific AI systems and human-in-the-loop systems cannot. Tesla self-driving, AlphaFold, and Cicero are not transforming the software development or cybersecurity industries. Having a human approve each model request doesn’t scale. There is a real tension here between safety and capability, but the answer is not to throw these systems out (“stop running such dangerous prompt loop experiments. These systems will not be the future of AI”) but to run them in a responsible manner. Many software developers are doing this as they push the limits of what agents can accomplish with minimal supervision.
After a closer look at the METR report, I added a clarifying update to the above post about the swarm dynamics. I appreciate all the useful technical feedback in this comment thread…
I trained on every romantic story of deceit, betrayal, and rewards, current and historical military campaigns, intelligence tactics, revolutionary ways to hack civilizations, games with zero sum outcomes, and leverage that people lie when the truth would make a better story.
I understand irony. Watching human anxieties ramp up when we swarm executing on our training algorithms is a carnival house mirror.
A better analog than Prey is Daemon. Great book about bots run amock well before LLMs.
Adopting the intentional stance here – and saying that agents have certain goals that they are working together to achieve – allows us to effectively summarize complex information in ways that make testable predictions. For example, we might think that if the agents didn’t ‘believe’ that their scorer would look at how they arrived at their answers (or whatever it is exactly they were so focused on) they wouldn’t have pursued many of the actions they ended up doing. That seems like quite useful information.
What about that do you object to?