In the AI doom discourse, I am considered an optimist. What that means is I think there’s only a tiny chance we all die and a very big chance that millions of people die, basic digital infrastructure becomes unusable and crashes the economy, and terrorist groups can use superintelligence for mass murder and mayhem.
Because of my familiarity with effective altruist circles, I’ve long been aware of a subculture concerned about existential risk caused by AI. But over the past several weeks, many more people in my life have become aware of this concern, particularly after Jacob Coxon publicly announced his resignation from Anthropic. He followed this with a media tour during which he argued that the leading AI labs are causing a 10% risk of human extinction — and many of his former colleagues agreed.
These are inherently terrifying and extraordinary claims. Anytime I really sit with them, I begin to feel a sense of dread and fear. Cultivating this dread on a mass scale seems to be the explicit goal of many AI safety advocates. Are they right?
Leadership at the nonprofit ControlAI has argued that “the awareness of extinction risk needs to be pervasive throughout society” and “the awareness needs to be specifically about extinction risk from superintelligent AI.” GuardRailNow, an AI safety organization that creates media about AI extinction risk, has laid out an explicit theory of change along the following lines: “Education leads to awareness, awareness leads to pressure, and pressure leads to regulation.” The organization argues that “when people understand the risks posed by unregulated AI they’re more likely to demand safety legislation from their elected leaders.”
This theory of change might be correct. There are times in history when awareness campaigns have yielded significant change, most notably the midcentury civil rights movements. Clearly, if you work on an underrated issue, you need to raise its salience among policymakers and politicians. One effective way of doing that is raising its salience among the general public.
But I’ve come to understand that when people imagine that they need public attention, in reality, they are often looking for institutional attention: securing funding, setting and enforcing standards, negotiating technical agreements, or updating regulations. It can be difficult or even impossible to separate public from institutional attention, but movements err when they treat public awareness as the sole or even main way to make change. In general, causing mass panic is more likely to result in counterproductive or nonexistent policy outcomes than it is to achieve the goals of a movement.
That’s because public concern does not necessarily translate into consensus on an appropriate response.
Fears of nuclear power plant meltdowns, particularly after the 2011 Fukushima disaster in Japan, prompted many to turn against nuclear energy as a source of stable, zero-carbon power. In Germany, policymakers accelerated planned shutdowns of reactors and reoriented the national grid toward coal generation and electricity imports. This was, of course, bad for climate change — but it also made the nation more reliant on Russian natural gas, which became a huge problem after Russia’s invasion of Ukraine. Spiking energy costs now may be abetting the rise of the far-right Alternative for Germany party.
In 2020, the two biggest issues in the U.S. were COVID-19 and police brutality. It’s hard to imagine that more activism could have increased their salience. And yet policy advocates have not been able to get critical pandemic preparedness policies passed by Congress. And while many states and localities passed various police accountability measures, no comprehensive federal police-reform bill passed either.
Another example of the counterproductive nature of mass panics is the media frenzy around “superpredators” in the 1990s. Journalists warned about a coming generation of remorseless teen criminals, and fears of this “new breed of vicious kids“ spurred a push to send violent arrested teenagers to adult courts. But this didn’t deter crime — in fact, it increased subsequent offending by 34%.
This mirrored an earlier push in the 1970s, which popularized “Scared Straight” programs to freak kids out about going to jail. Those programs, too, increased future offenses.
In today’s episode of The Argument podcast with my co-host Matt Yglesias, I argue that we don’t need to buy into the existential threat of AI in order to make real progress on AI safety. Many AI safety advocates believe that it’s critical not just to raise awareness but also to get people to agree with them on the exact way that AI could be harmful. I think they should let go of this dogmatism and embrace the spirit of overlapping consensus.
You don’t need to believe in the possibility that misaligned AI will turn us all into paper clips to agree it would be bad if terrorists got their hands on a technology that makes executing biological attacks easier. You don’t need to believe in AI’s ability to access the world’s nuclear arsenal in order to believe it would be bad if hackers destabilized the world economic system by making digital banking unsafe.
Matt is much less optimistic than I am; he worries that not taking extinction risk seriously will prevent us from taking the steps necessary to safeguard against AI.
He could be right, but I think humans tend to believe other people have to agree with our reasons more than is actually necessary. I don’t need you to believe in God to agree that I should be free to go to church. I don’t need you to agree with me about what a woman is in order for us to agree someone shouldn’t be fired from their job for being trans. And I don’t need you to agree with my forecast that AI could kill millions of people to agree that we should probably do something about this technology that is already committing crimes.
When President John F. Kennedy made the case for the 1963 Limited Test Ban Treaty, he warned about nuclear war, radioactive fallout, and the spread of nuclear weapons. The Joint Chiefs of Staff were much more concerned about maintaining America’s military advantage: They wanted underground testing to continue and the U.S. to remain ready to resume atmospheric tests if necessary. But they ultimately supported the treaty with those safeguards in place. They didn’t have to agree about which danger mattered most to agree on something worth doing.
I have a knee-jerk skepticism of the efficacy of a politics of fear. People and powerful institutions are already primed to care about biological attacks, keeping the financial system secure, and protecting digital infrastructure. While high-salience activist efforts are more memorable — for the obvious reason that they specifically operate through gaining public attention and therefore have a chance to embed themselves in our collective memory — quieter feats of careful cooperation often solve big problems.
Though it’s in vogue to dunk on international cooperation, I think doing so leads people to doom about how even hostile states can find the common ground necessary to tackle seemingly intractable issues.
In 2011, the world eradicated rinderpest — a disease that devastated cattle herds throughout Europe, Asia, and Africa — through coordinated vaccination, disease surveillance, and information campaigns. And by 2021, a UN-led partnership had helped complete a global phaseout of leaded gasoline by helping governments update regulations and helping refineries make unleaded fuel. This effort has been estimated to prevent more than a million premature deaths annually.
The examples go on and on: international weather-data sharing, the U.S.-Soviet satellite search-and-rescue system, the Svalbard Global Seed Vault, the spread of iodized salt, and much more.
Matt and I argue over whether AI safety will be the next collective action problem that humanity triumphs over or the one that finally brings us to our knees.
The Argument. Libbing out.
Paying subscribers can find the transcript for this post beneath the paywall.
WATCH THE EPISODE ON YOUTUBE HERE
New episodes post every Thursday.
For an ad-free version and full transcript, subscribe at TheArgumentMag.com.
Corrections and clarifications:
Around 0:24:40, Jerusalem describes an Australian man who accidentally got an AI to hack into a gym’s software, but she says she did not research the details. This did happen, albeit with some different circumstances than described.
Around 0:40:00, Jerusalem describes a situation in which the U.S. “accidentally sent missiles to the USSR,” but the USSR correctly perceived that this was not a true attack. The USSR’s system did alert it to attacks from the U.S. on Sept. 26, 1983, but this was an error of its equipment, not real missiles.
Around 0:54:20, Matt objects to DeepSeek’s assertion that China has eight political parties, saying he doesn’t “totally know what that’s a reference to.” While China is a single-party state, it does have eight subordinate “satellite parties,” which “accept the leadership of the [Chinese Communist Party] as a precondition for participation,” according to the Australian Strategic Policy Institute.
Show notes:
If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, book by Eliezer Yudkowsky and Nate Soares, referenced by Matt: Goodreads page
“AI as Normal Technology,” article by Arvind Narayanan and Sayash Kapoor, referenced by Jerusalem: Knight First Amendment Institute at Columbia University essay
“Our framework for reporting model misalignment”: OpenAI misalignment report
“AI 2027,” prediction by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, and Romeo Dean, referenced by Jerusalem: AI 2027 site
“Nicholas Decker In Hell,” article by Scott Alexander, referenced by Jerusalem: Astral Codex Ten article
“Gym rat asks AI agent to book him a class, it hacks a waitlist API to bump him up the list,” article by Brandon Vigliarolo, referenced by Jerusalem: The Register article
“Why humanoid robots won’t catch up to human workers any time soon,” article by Kai Williams, referenced by Jerusalem: Understanding AI article
Silent Spring, book by Rachel Carson, referenced by Jerusalem: Goodreads page
“How NIMBYs hijacked the climate movement,” podcast episode about Silent Spring: The Argument podcast episode
Elon Musk tweet referenced by Matt:
Peer Review: “Meetings,” article by David J. Deming, Katrine V. Løken, Alexander Willén, and Yaling Xu: NBER working paper
Films referenced:
Ferris Bueller’s Day Off (1986)
Terminator 2: Judgment Day (1991)
The Devil Wears Prada (2006)
Maybe we should remember how we've dealt with something else that threatened to exterminate humanity. Nobody in the Fifties or the Sixties or even the Eighties ever imagined that we could go the eighty years since Nagasaki without a single nuclear weapon being used by anyone, anywhere. And yet that's what happened. (And that's with nukes in the hands of such sane, reasonable people as Stalin and Mao and Kim, not to mention Nixon and Trump and Putin and Netanyahu.) If we could get nuclear weapons (which, unlike AI, are only useful for killing or threatening to kill massive numbers of people) under control, maybe that could inspire us to believe that humanity is capable of doing the same with AI.
Strongly agree that the politics of fear are not the right way to go almost ever. What does seem true though is that often the worst-case scenario is necessary to achieve salience. People often point out the climate movement as a failure case, which is very odd to me as absolutely tons and tons and tons and tons of focus and action and international/corporate coordination etc. has gone towards climate mitigation — directly downstream of salience increased by worst-case-scenarioism.