Over the last several months, the two leading frontier AI labs have shown some brazen behavior.
It started with a series of carefully planned announcements and reports from OpenAI that try to establish how unnerving and powerful (not to mention felonious) their LLM-powered agent systems have become.
They then handed the baton to Anthropic, whose employees began publicly debating the exact probability that these technologies would lead to human extinction. They delivered this all with an eerie calmness that conveyed a nihilistic inevitability about their work.
The ground suitably softened, the campaign crescendoed with Anthropic CEO Dario Amodi releasing a letter, titled “We Must Pace the Frontier,” in which he enumerates all the harms his own company’s research might cause, and then, instead of apologizing and promising to stop, concludes that catastrophe can only be avoided if we “build the technology in the right way,” which involves – surprise, surprise – the government slowing down potential competitors and allowing the labs to take the lead in advancing the relevant tech. OpenAI CEO Sam Altman quickly tweeted his support for this brave plan.
Internally, these labs have long embraced this style of messianic thinking, casting themselves as humanity’s only hope against the defied powers of superintelligent AI. This summer, in some sense, was their attempt to induct the rest of us into this long-held ideology.
It didn’t work.
Instead of inspiring the masses to stand up and applaud these engineers’ heroic rationalism, it made us instead start to ask, “What the hell is going on over in those labs?”
To that end, last Thursday I published an op-ed in The New York Times calling on Congress to begin a public fact-finding mission to uncover what research projects OpenAI and Anthropic are conducting, how they’re conducting them, and for what purposes.
I suggested three key areas to explore with this questioning:
- We must move past vague discussions of “AI” and instead isolate the specific types of systems that are creating problems. The frontier labs want us to think of AI as a singular technology on an inevitable, fixed trajectory. In reality, most of the problems in recent months result from a narrow band of incautious experiments, conducted mainly by the frontier labs. They need to justify why they’re running them.
- We must examine the internal safety procedures surrounding these research efforts. OpenAI revealed a long series of unauthorized hacking attacks by their autonomous agents. Why weren’t these efforts stopped after the first such incident was revealed? Why not pursue criminal liability for them running systems they knew would likely commit crimes?
- We have to investigate the role of apocalyptic futurist ideologies in the decisions being made by the frontier labs, both about what research to pursue and how fast to pursue it. I’m increasingly concerned that OpenAI and Anthropic, in particular, are becoming increasingly reckless because they believe collateral damage is justified in a race to redeem humanity. (For more on these beliefs and their connections to the labs, see: this and this and this, all published in the Times in just the last several weeks.)
“We need to stop letting a small number of private companies, acting and talking in increasingly erratic ways, dictate how we’re supposed to feel about A.I.,” I concluded. “We’ve heard what they want to say; now it’s Congress’s turn to step in on our behalf to find out what’s really going on.”
Love these articles, and thank you for them. I suspect a typo in the phrase “the defied powers of superintelligent AI” was meant to be “deified”.
Jensen Huang’s interview with Ezra Klein put this clearly: Labs asking for government to slow them down is abdication of responsibility and blame. They are responsible to make their products safe. No one is pressuring them to release unsafe products.
They need to likely grow their investment and headcount 10x, and 80% should be safety and testing.
And finally, if the labs are unable to stop releasing unsafe products, they should probably be shut down
Everyone should listen to that interview. It’s a window into how these folks think and it’s frankly shocking. In Huang’s case, I assume his odd answers and lack of concern over AI safety are the result of being compromized to the tune of a 4 trillion dollar market capitalization. His reasoning lacked coherence and felt uninformed (especially when it came to the issue of alignment and safety). Huang seems completely unwilling to entertain that intelligent systems (even generally intelligent systems) are anything but “software systems”.
I have no idea why he agreed to do that interview. It strongly reinforces the concluding statement in the essay above.
Nope! They want to establish a regulatory environment which they find relatively easy to navigate (having had a hand in architecting it) yet which creates tall barriers-to-entry for against competitive players, like open models and whatever. Nothing to do with abdicating responsibility and everything to do with desperately trying to ensure their self-preservation. Though, of course, who wouldn’t want a free side dish of responsibility abdication with their order of self-preservation.
I agree with you on the labs needing to do more safety work.
However, the basic argument for why the labs keep pressing forward goes like this:
– They believe that AI is inevitable in the sense that if they don’t work on it, someone else will shortly.
– They believe that other companies building AI will act less responsibly than them.
– Therefore, they need to keep going as long as there aren’t regulations stopping everyone, because they don’t want someone less responsible to become the market leader.
Does that argument make sense?
Here is a different way of putting it. AI companies believe that there is uncertainty about whether or not AI goes well. (For sake of argument, assume that they think there’s an 80 percent chance of it going well and a 20 percent change of it going wrong.) That number might be dependent on how responsible and safety-conscious the company doing it is, where a better run company thinks of themselves as having a better chance of building the technology safely than others. That’s the fundamental reason why it makes sense for Dario to race ahead while simultaneously warning about the consequences and asking for regulation. The obvious thing that would happen if Anthropic stopped is that a different company (likely a less responsible one) would keep going and eventually become the well-capitalized market leader, which you don’t want if you think that you’re more responsible than the other companies further behind you.
For hs: Jensen Huang and Dario Amodei have very different mindsets about this, and neither of their beliefs are representative of the other. Jensen Huang’s mindset is, “the companies themselves are responsible for making their products safe”. While this is true, it doesn’t remove the need for regulation or outside investigations, because we don’t treat any other industries this way (everything from Drug companies to Sandwich shops all have outside inspectors, laws against releasing unsafe products, and so on), and therefore regulation here is nothing abnormal. Jensen Huang is, as far as I know, very anti-regulation, while Dario Amodei is very pro-regulation. So my take would be, “yes, the labs should take responsibility for making their systems safe, but we also need some kind of regulation on top of that.”
If Anthropic announced that they were shutting down tomorrow, all the investment and talent from them would just go somewhere else, which is why making a law is the only good way to stop this.
Thank you Cal. A lot of these guys are just selling snake oil. At some point it can mess up a system and nobody knows what to do.