Hello, and welcome to TechScape. I’m Blake Montgomery, US tech editor at the Guardian. Today in tech, we’re discussing the past week’s all-consuming apoplexy over AI safety.
Why did this warning about the AI apocalypse break through when so many others didn’t?
Less than a week ago, a young Anthropic researcher announced his resignation from the AI giant on X, in a post that has shaken Silicon Valley, Washington DC, and even Beijing.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” wrote Jacob Coxon.
Since then, the post has been viewed 171m times, per X’s metrics. Why did it travel so far?
It’s not the first time AI insiders have issued dire warnings. Every few months for the past several years, an AI researcher has resigned in a moment of crisis, proclaiming the impending apocalypse. His proclamation comes at a unique political confluence, a moment of high-strung public fear and rage.
Loudly declaring that your peers in Silicon Valley are failing at AI safety is as old as the companies themselves. OpenAI’s board fired Sam Altman over safety concerns in 2023. Dario Amodei split from OpenAI five years ago with the aim of founding a “safety-first” lab. In 2015, Altman and Elon Musk founded OpenAI as a counterweight to Google’s approach to AI. Musk sued Altman for failing to keep a fundamental safety promise and lost.
Major AI researchers have made second careers of predicting doom. Geoffrey Hinton, a Nobel laureate known as “the godfather of AI”, said as far back as 2023 that the technology could cause human extinction within three decades. He’s since shortened his timeline. He resigned from Google for reasons similar to those of Coxon. Hinton has been on the speaking circuit ever since, prophesying doom.
Hinton’s resignation even inspired a significant response from lawmakers. In the US, Joe Biden issued an executive order mandating safety testing by AI companies. Congress summoned AI CEOs for a grilling, though that resulted in little legislative action. In the European Union, the EU Artifical Intelligence Act passed in part due to a strong reaction to Hinton’s warnings.
And yet, here we are again, burdened with the feeling that little meaningful regulation is preventing harm by AI.
So why is Coxon’s apocalyptic proclamation percolating deeper?
First, there’s the messenger: Coxon’s call for greater safety is credible by several metrics: He had insider access to both Anthropic and OpenAI, where he worked for five years. He resigned before he would have received equity in Anthropic, demonstrating the personal cost of speaking out. And his colleagues, still on Anthropic’s payroll, corroborated his assertion that many AI researchers believe there’s a possibility that AI will bring about human extinction, with one co-worker gauging that outcome as more than 10% likely.
Recent incidents in the AI field have made apocalyptic claims more credible. A swarm of AI agents operated by OpenAI went rogue, broke containment and hacked into multiple sites and organizations during safety testing with startling coordination earlier this year. On Thursday, Anthropic detailed how criminals and rogue scientists had attempted to use Claude to make bioweapons.
“The reason this time is different is that out-of-control AI is not in the future, but in the past. People have seen evidence of it happening,” argued Max Tegmark, co-founder of the Future of Life Institute, which has been funded with hundreds of millions of dollars from the billionaire founder of the cryptocurrency Ethereum. (Tegmark’s organization already in 2023 called for a slowdown of AI development, which at the time provoked a rejection from Amodei.)
“If you had tried to get people worried about nuclear winter in 1942, good luck. People would ask whether a nuclear weapon was even possible,” said Tegmark. “It was obvious to physicists then, but you couldn’t have freaked out normal people. It was all theoretical. With AI, it was all theoretical until this happened. Now the debate over whether you can lose control is over.”
Then there’s the context. The general feeling towards technology in the US is kindling-sensitive, and claims of impending doom are fiery. The country is in the throes of a multifaceted backlash to technology. People outside major tech hubs feel imposed upon physically by data centers and figuratively by AI’s takeover of the workplace.
Looming over the country are tech billionaires – in control of vast wealth, global communication channels – throwing their weight around in US politics with unmatchable campaign contributions.
“It’s been at least over a year of steadily increasing data center pushback that has made being pro-AI regulation the new common sense, and it certainly has made being pro-AI industry a very toxic brand,” said Amba Kak, co-executive director of the AI Now Institute and a former senior adviser on AI for the US Federal Trade Commission.
There’s the fact that the companies involved have responded vigorously to Coxon. Their pledged commitments have details and heft.
“It allowed for companies like Anthropic and others to position themselves immediately as the primary experts in how to deal with these fears. How to solve problems that, by every account, they have created or are best positioned to prevent,” Kak said.
Two closing thoughts. In a sign of how broad the crisis of confidence in AI has become: Strange bedfellows have partnered up in response. The Future of Life Institute is hosting a rally today in Washington promoting AI safety with headliners Bernie Sanders, the progressive senator from Vermont, and Steve Bannon, former chief of staff to Donald Trump, who have been sworn enemies on other issues.
And finally: Why is the AI industry so full of men – only men in AI have expressed these fears of human extinction – in the throes of a crisis over the extinctive potential of their work?
“These people certainly believe what they’re saying. It’s not a fringe view,” said Shakeel Hashim, writer of the Transformer newsletter on AI. “The best way I’ve had it explained to me is that we’re trying to build something that’s smarter than us in the way we’re smarter than monkeys. Our existence hasn’t gone great for monkeys; AGI’s existence probably won’t go great for us.”
Read more about the great AI freakout
AI CEOs say they need to slow the pace of development. But will they?
OpenAI urges UK lawmakers to rein in technology amid growing safety fears
Europe must build own AI or risk getting cut off by US or China, says ECB’s Lagarde
Trump attacks ‘sick conspiracy’ against AI as tech stocks slide
Microsoft proposes limits on its AI with code of conduct amid safety debate
Make sense of the crisis
