A couple of weeks ago, Ezra Klein interviewed AI researcher Eliezer Yudkowsky about his new, cheerfully-titled book, If Anyone Builds it, Everyone Dies.
Yudkowsky is worried about so-called superintelligence, AI systems so much smarter than humans that we cannot hope to contain or control them. As Yudkowsky explained to Klein, once such systems exist, we’re all doomed. Not because the machines will intentionally seek to kill us, but because we’ll be so unimportant and puny to them that they won’t consider us at all.
“When we build a skyscraper on top of where there used to be an ant heap, we’re not trying to kill the ants; we’re trying to build a skyscraper,” Yudkowsky explains. In this analogy, we’re the ants.
In this week’s podcast episode, I go through Yudkowsky’s interview beat by beat and identify all the places where I think he’s falling into sloppy thinking or hyperbole. But here I want to emphasize what I believe is the most astonishing part of the conversation: Yudkowsky never makes the case for how he thinks we’ll succeed in creating something as speculative and outlandish as superintelligent machines. He just jumps right into analyzing why he thinks these superintelligences will be bad news.
The omission of this explanation is shocking.
Imagine walking into a bio-ethics conference and attempting to give an hour-long presentation about the best ways to build fences to contain a cloned Tyrannosaurus. Your fellow scientists would immediately interrupt you, demanding to know why, exactly, you’re so convinced that we’ll soon be able to bring dinosaurs back to life. And if you didn’t have a realistic and specific answer—something that went beyond wild extrapolations and a general vibe that genetics research is moving fast—they’d laugh you out of the room…
But in certain AI Safety circles (especially those emanating from Northern California), such conversations are now commonplace. Superintelligence as an inevitability is just taken as an article of faith.
Here’s how I think this happened…
In the early 2000s, a collection of overlapping subcultures emerged from tech circles, all loosely dedicated to applying hyper-rational thinking to improve oneself or the world.
One branch of these movements focused on existential risks to intelligent life on Earth. Using a concept from discrete mathematics called expected value, they argued that it can be worth spending significant resources now to mitigate an exceedingly rare future event, if the consequences of such an event would be sufficiently catastrophic. This might sound familiar, as it’s the logic that Elon Musk, who identifies with these communities, uses to justify his push toward us becoming a multi-planetary species.
As these rationalist existential risk conversations gained momentum, one of the big topics pursued was rogue AI that becomes too powerful to contain. Thinkers like Yudkowsky, along with Oxford’s Nick Bostrom, and many others, began systematically exploring all the awful things that could happen if an AI became sufficiently smart.
The key point about all of this philosophizing is that, until recently, it was all based on a hypothetical: What would happen if a rogue AI existed?
Then ChatGPT was released, triggering a general vibe of rapid advancement and diminishing technological barriers. As best I can tell, for many in these rationalist communities, this event caused a subtle, but massively consequential, shift in their thinking: they went from asking, “What will happen if we get superintelligence?” to asking, “What will happen when we get superintelligence?”
These rationalists had been thinking, writing, and obsessing over the consequences of rogue AI for so long that when a moment came in which suddenly anything seemed possible, they couldn’t help but latch onto a fervent belief that their warnings had been validated; a shift that made them, in their own minds, quite literally the potential saviors of humanity.
This is why those of us who think and write about these topics professionally so often encounter people who seem to have an evangelical conviction that the arrival of AI gods is imminent, and then dance around inconvenient information, falling back on dismissal or anger when questioned.
(In one of the more head-turning moments of their interview, when Klein asked Yudkowsky about critics–such as myself–who argue that AI progress is stalling well short of superintelligence, he retorted: “I had to tell these Johnny-come-lately kids to get off my lawn.” In other words, if you’re not one of the original true believers, you shouldn’t be allowed to participate in this discussion! It’s more about righteousness than truth.)
For the rest of us, however, the lesson here is clear. Don’t mistake conviction for correctness. AI is not magic; it’s a technology like any other. There are things it can do and things it can’t, and people with engineering experience can study the latest developments and make reasonable predictions, backed by genuine evidence, about what we can expect in the near future.
And indeed, if you push the rationalists long enough on superintelligence, they almost all fall back on the same answer: all we have to do is make an AI slightly smarter than ourselves (whatever that means), and then it will make an AI even smarter, and that AI will make an even smarter AI, and so on, until suddenly we have Skynet.
But this is just a rhetorical sleight-of-hand—a way to absolve any responsibility for explaining how to develop such a hyper-capable computer. In reality, we have no idea how to make our current AI systems anywhere near powerful enough to build whole new, cutting-edge computer systems on their own. At the moment, our best coding models seem to struggle with consistently producing programs more advanced than basic vibe coding demos.
I’ll start worrying about Tyrannosaurus paddocks once you convince me we’re actually close to cloning dinosaurs. In the meantime, we have real problems to tackle.
If Gen Z keeps dumbing down via TikTok and can’t read long books, then society will be a little dumber than AI, making AI seem a littler smarter than people. We may reach super-UNintelligence in people before we reach super-intelligence in AI.
Cal, I need the podcast timestamp for when Yudkowsky dismisses you as a Johnny-come-lately 😂 I need to hear what his on-the-defensive butthurt tone was like!
Here’s the link with the timestamp: https://youtu.be/2Nn0-kAE5c0?si=4lykPqCopKlSWbCX&t=3606
I think Cal misunderstood what Eliezer was saying. Copying from my other comment (which is maybe caught in the spam filter??):
Eliezer was saying that there are people who only recently heard of AI, and don’t realize that there is such a thing as non-LLM AI. Their perspective is: “if LLMs can’t scale to superintelligence, then, well, I guess superintelligence is impossible.” (I have definitely met lots of people like this!) Whereas Eliezer has been around long enough to know that there can exist more than one AI paradigm, and if LLMs can’t scale to superintelligence, then AI researchers will look for another paradigm that can.
You are an excessive LessWrong contributor. Perhaps you are too wrapped up in the community to respond to this post?
I agree with you that LLMs are not part of the path to superintelligence. I also agree with you that superintelligence is not imminent (meaning, in the next 5 years, as e.g. Dario Amodei and Sam Altman and others have been saying).
But I strenuously disagree that this constitutes a reason not to talk about superintelligence!
(Context: I’m an LLM-skeptical superintelligence safety researcher. I haven’t listened to that podcast yet, sorry if anything here is missing the point.)
If you (Cal Newport) believe that superintelligence will never ever happen because it’s fundamentally impossible, like time-travel, then I wish you would take a stand and say that loud and clear, and we can have a discussion. I would strongly disagree on the grounds that if human brains exist, and human brains can do things wildly beyond LLMs, and they work by (yet-to-be-invented) algorithms, not magic; and putting those algorithms on chips would allow dramatic speed-ups and scale-ups that would constitute superintelligence even without “recursive self-improvement”.
Or if you believe that superintelligence is possible, but you are overwhelmingly confident that it is more than 20 years away from being invented, then I again wish you would take a stand and say that loud and clear, and again we can have a discussion. I would say that this is way overconfident; I think the right stance is “superintelligence might or might not happen within 20 years, who knows, hard to say either way”. Technological forecasting is very hard, 20 years is way more than enough time for multiple new AI paradigms to be invented and extensively developed (think of all that’s happened in AI since 2005!), many LLM-SKEPTIC domain experts (e.g. Chollet, LeCun, Sutton) nevertheless guess that superintelligence is plausible or even likely within 20 years, and so on.
Or if you believe that superintelligence might or might not happen within 20 years, but you’re puzzled by the fact that people are talking about it, then I claim you are not thinking about the situation very clearly. People routinely think 20 years out in every other aspect of life—family planning, retirement planning, infrastructure planning, institution planning, etc. Indeed, in climate change, people talk about bad things that might or might not happen in 2100! If you really actually expected in your gut that superintelligence might or might not happen within 20 years, then it would be a huge part of how you think about your life and the world.
For my part, I think Alan Turing and IJ Good and Norbert Weiner etc. were absolutely right to be talking about and worrying about superintelligence, not because they thought it would happen in 5 years or 20 years or even their lifetime, but because they knew it was going to happen eventually, and because they recognized its profound consequences.
As for your analogy, if tens of thousands of experts were spending billions of dollars trying to clone Tyrannosaurus; if almost everyone agreed that cloning Tyrannosaurus was theoretically possible and would happen sooner or later, and it’s just a question of how soon; if a bunch of experts said it would happen in 5 years but so-called skeptical experts said “that’s crazy hype, it’s probably more like 15 years”; and if a large number of dinosaur cloning experts, including some of the most esteemed founders of the dinosaur cloning field, were saying that Tyrannosaurus cloning would quite possibly lead to outright human extinction …
Then yeah, in that hypothetical, people should absolutely 100% be giving talks on electric fences for cloned Tyrannosaurus! Right?
Separately, you misunderstood Eliezer’s “Johnny-come-lately” comment in the podcast. He was saying that there are people who only recently heard of AI, and don’t realize that there is such a thing as non-LLM AI. Their perspective is: “if LLMs can’t scale to superintelligence, then, well, I guess superintelligence is impossible.” (I have definitely met lots of people like this!) Whereas Eliezer has been around long enough to know that there can exist more than one AI paradigm, and if LLMs can’t scale to superintelligence, then AI researchers will look for another paradigm that can.
Happy to elaborate on any of these points.
Superintelligence could be possible, but a lot of existential threats are also possible within 20 years as well.
At present, these technologies don’t demonstrate the capability to become superintelligent. Superintelligence is purely theoretical. Talking about superintelligence as if it WILL happen, when the reality is that there’s no proof of that, is just posturing that ends up making AI seem more important than it actually is.
I know a lot of people are invested in this issue, and that’s fair. I just think there’s other aspects of AI we should be more worried about…
Thanks! But I claim that you’re failing to think clearly about rational planning under uncertainty.
There is no proof that superintelligence will definitely happen within 20 years; this is true. There is also no proof that China will definitely invade Taiwan within 20 years. But it’s quite plausible. So we should be thinking about and planning for that “theoretical” possibility. Right?
For that matter, there is no proof that, if you put the space heater next to the curtains, it will start a fire. It’s not guaranteed. But it’s plausible! So you should move the space heater somewhere else! 🙂
The idea that we should work to mitigate possible risks only when we have “proof” that those risks will definitely manifest, is crazy, and is not the standard that you would apply in any other domain. I bet there are things in politics that you’re worried about happening, where you lack any “proof” that they will definitely happen. “Theoretically”, Donald Trump could openly defy the Supreme Court, kicking off a constitutional crisis etc. Or pick any of a million other things that people talk about in the news. You mention “other aspects of AI we should be more worried about”. What are they, and do you have “proof” that they will definitely happen? I expect not. For example, I myself am worried about polarization, deepfakes, and people getting addicted to virtual friends. (I’m MUCH MORE worried about superintelligence! But I am also worried about lots of other things. The world has more than one problem.) But my worry is mainly that the situation regarding deepfakes, virtual friends, etc., will be different and (even) worse in say 2028 than it is today. So is now back to being a “theoretical” concern that lacks “proof”?
Anyway, the best thing to do when we need to act under uncertainty (which we always do) is to do what the superforecasters do: assign numerical guesses to uncertain and unprecedented future events. Yes it’s hard, but we have to do the best we can with the limited information we have—there is no other reasonable option.
For example, what’s the probability that China will invade Taiwan within 20 years? In a sense, we don’t know; it would be a one-off unprecedented event. But in another sense, we absolutely do know something. The probability is obviously more than 1%. It’s obviously less than 99%. We can read the Superforecasting book and use those techniques to try to narrow it down further. Right? There are prediction markets on this topic, and there are reasons to believe them.
By the same token, ASI is a potential future technology. We have abundant historical evidence that once-“theoretical” technologies, things that seemed like sci-fi at the time, can wind up getting invented. We know that ASI is theoretically possible, and we know that many people are trying to invent it. We know that domain experts disagree about how soon this will happen, but that many or most of them, including those skeptical of LLMs, think it’s quite possible within 20 years. Under those circumstances, if you start assigning a probability of 95%+ that ASI is more than 20 years away, then I think you’re wayyy overconfident. I think 50-50 is a much more reasonable guess. If you say 20%, or 80%, OK sure, this is in the range where reasonable people can differ. But if you treat this as a ludicrously unlikely Pascal’s Wager thing, then I think you’re way off. “ASI in 20 years” is something that we should view as somewhere between “unlikely but plausible” and “likely but not a sure bet”. It’s definitely the kind of thing we should be thinking hard about and planning for.
As a side-note, I disagree that there are “lots” of other ways besides superintelligence that humanity could be extinct within 20 years. What did you have in mind? Nuclear war and climate change could be very bad, but all the serious analyses that I’ve seen say that they wouldn’t cause human extinction within 20 years. Asteroids and supervolcanoes might, but they’re extremely unlikely (1 in 5,000,000 and 1 in 50,000 respectively) to happen in the next 20 years. (We know this because we can calculate the historical base rate.) Engineered pandemics are a candidate IMO, but that’s just one other example, whereas you said “lots”.
I do think there’s more than one problem in the world, and if someone is trying to prevent engineered pandemics, or mitigate climate change, etc., then good for them.
(…Oh, I guess you’re probably using “existential risk” to mean “big problem” rather than “every human has stopped breathing, or similar”. I agree that there are lots of “big problems” in the world.)
Thank you for the thoughtful (and, in my opinion, convincing) answer Steven. It’s not a certainty that we’ll reach superintelligence at some point, but it’s plausible enough (like pandemics, asteroids or nuclear war) to warrant extremely serious discussions, at the highest levels.
You’re on LessWrong a lot. Maybe you’ve drunk too much koolaid to be able to respond to these thoughts?
the existential threat posed by AI is not superintelligence, but the parade of radicalizing AI-generated slop on everyone’s feeds.
Touching grass and deleting social media literally changed my life. I can’t emphasize enough how these conversations about a “superintelligent AI” annoy me. If anything, these conversations are desperately trying to keep AI relevant, important, and inevitable. They turn people’s eyes away from the limitations of this technology. Keeping people looking towards the future keeps them from looking at the present.
The one thing I keep being reminded of is that if we believe man evolved from lesser forms of life, than the same logic would assume we could be outperformed by our own creation.
However, while popular in ideology, neither of these have been observed in real life, but are dependent on many assumptions that are not scientifically provable by repetition. Similar to the possibility of cloning a T – rex, it sounds cool, but it cannot be repeated in real life no matter how many people work on it and for how may years.
In my opinion the capping off of AI’s abilities is just another reminder that an intelligent creation is proof of an more intelligent creator in both AI and science.
Although I think ASI is unlikely to arise in the coming decades, if it is sufficiently plausible to arrive within our lifetimes, then it is important to consider the ramifications if it occurs, especially when they are sufficiently grave. Nuclear weapons and gene editing were once thought to be impossibilities but now exist and pose a threat to humanity. If no one worked on nuclear weapons or gene editing bacteria (for good or bad), then sure, analyzing there danger is nothing more than sci-fi. But if billions of dollars are being put into developing these technologies in the hopes of achieving super intelligence, then we should consider the ramifications of their goals.
The underlying disagreement to me seems like you don’t believe it is possible for AI to truly be “intelligent”, and this is an interesting yet difficult conversation because no one really knows what intelligence is. But how intelligent does AI need to be to be dangerous? It already seems to be having negative effects with generated content and bots flooding social media. How much worse can it get if these technologies where, say 50% more “intelligent”? There is a continuum of dangers AI creates as it gets progressively more intelligent, and its important to analyze and discuss them at every step rather than just blindly continue to invest.
Hi Cal, love your work. I’m a computer engineer, and I currently work in cyber security. I’m very skeptical of the current AI mania. We all know its done great things in protein folding and playing go, but it’s still a huge leap to general intelligence. I think LLMs are really, really good search engines, and I’d be very careful about using them for most functions beyond that.
However! I listened to the audiobook of Yudkowsky’s and Soares’s book two weeks ago and I think the salient point is that it doesn’t matter how it’s built, or even if it can’t be built. The point is where are we going to be if we find it can be built? This is a very, very valid question. Better to think about these things now, and even try and get the treaties in place.
Imagine if a similar and successful effort had taken place in advance of the first nuclear weapon being produced, or at least the first thermonuclear weapons?