Superintelligence is a Fairy Tale. But Chasing It Can Still Cause Harm.

Yesterday, multiple Anthropic employees casually announced online that they thought the types of AI they’re building could kill billions of people. Not in some distant hypothetical future, but in the next few years. Understandably, this created a tsunami of media attention and anxious discussion.

I added to this debate with a ​brief essay​ of my own. I fear, however, that in the Sturm und Drang of that charged news cycle, some of my conclusions might have been hard to follow. So, I thought it would be useful to publish three short follow-up points that summarize my core claims more directly…

Point #1: Anthropic and OpenAI are not able to create superintelligent AI.

The Anthropic employees who spoke out yesterday believe the force that might wipe out humanity will be “superintelligent” AI systems that are much too capable for us to control. Let me be clear: We have no idea how to create such machines. We don’t even know if they’re possible. And if they are possible, we are many, many key technical innovations away from getting there. (Simply scaling LLMs won’t be nearly enough.)

The LLM labs attempt to sidestep this reality ​by claiming​ their models will figure this all out on their own, programming better versions of themselves that will be far more capable than anything us measly humans can conceive. But this is just a warmed-over version of the recursive self-improvement trope that Silicon Valley futurists have been peddling ​since the 1960s​; a rhetorical crutch that lets you somberly discuss sci-fi scenarios without actually engaging with relevant technical details.

Point #2: But their obsession with superintelligence can still create real harm.

When you believe,​ as many at OpenAI and Anthropic reportedly do​, that humanity’s fate rests on winning a race to create the most powerful technology ever, you tend to care less about the damage you inflict along the way.

This is what happened with the HuggingFace hacking attacks from this summer. The experiments OpenAI ran were reckless. They took a relatively common technology called an LLM-powered agent, which is used with minimal issues by millions of software developers every day, and ​then added​ a bunch of dangerous features, removed restrictions, and released a whole mess of them into poorly fenced digital wilds. Why? They were desperate to make as much progress as quickly as possible on a hacking test that they felt was an important milestone en route to superintelligence.

This is the equivalent of deploying a fleet of experimental self-driving cars onto the interstate, hoping that at least one will avoid crashing long enough to win a navigation challenge.

In other words, this behavior was negligent. But they likely justified it because it served the far grander goal of summoning a digital God. (Clearly, there may be additional motivations at play here as well. Given OpenAI’s interest in an IPO, for example, there might have been pressure to do well on this particular benchmark to establish their model’s power. But the more I talk to sources connected to these companies, the more I’ve come to believe that their commitment to futurist ideas is what really drives a lot of their more irresponsible behavior.)

Ultimately, I think the HuggingFace attack is a good representation of the types of near-future harm we should actually be fearing if the LLM labs continue to operate as they currently are: that is, more cyber mayhem than catastrophic risk.

But just because these impending harms aren’t nearly as scary as the extinction events touted by the true believers, they’re still plenty worrisome. More importantly, they’re entirely avoidable!

Which brings me to my final point…

Point #3: AI progress does not have to be this scary or dangerous

First things first, it’s important to emphasize something that many media figures and commentators seem to be missing: AI is not the same thing as LLMs. Many of the most impressive AI systems that exist today – including those that beat humans at games, win Nobel prizes for studying proteins, and self-drive cars through crowded city streets – have very little to do with LLMs. Slowing down OpenAI and Anthropic doesn’t mean that you’re stalling AI progress in general. You would only be impacting the subset of AI research that focuses on hyper-scaled LLMs as a central component.

But even when we do focus specifically on LLM-based AI research, none of the harms I described above are unavoidable. They’re instead side effects of reckless experiments motivated by the pursuit of superintelligence.

If the LLM labs simply acted more responsibly, focusing on producing useful products instead of summoning an AI messiah, and following common-sense research safety protocols instead of racing haphazardly toward some imagined utopia, we could be enjoying a steadily advancing AI industry promising tools that excite us, all without having to endure regular doses of dread and fear.

But due to their ideological commitments to superintelligence, these labs seem incapable of such reforms.

This last observation is what’s driving me to advocate for intervention. Those Anthropic employees absurdly implying that they’re willing to risk a digital holocaust to realize their futurist vision was my last straw. They’re far too weird for us to expect them to ever act reasonably.

Please don’t take this as a call for abandoning AI research or somehow ceding AI supremacy to China. Our problem is not with AI progress, but instead with the eccentric way this small group of companies is trying to achieve it. Fix that, and AI innovation can continue, free from all the doom and with much less potential for collateral damage.