In early October, The Argument’s editor Jerusalem Demsas and I attended a briefing at Anthropic’s headquarters in San Francisco that discussed AI-enabled bioweapons risks. I’ve never once attended a bioweapons briefing that didn’t make me feel a bit queasy; humans aren’t meant to spend too much time contemplating what the absolute worst people in the world might do if they had the power.
The worry is that, while right now you need to be a skilled virologist to know what components to order or how to grow and spread a bioweapon, AI tools might eventually make it possible for a rogue state, or a rogue group, or an individual, to do all this without virology expertise.
For a while, Anthropic tested whether Claude could assist a terrorist in developing a biological weapon in the obvious way: It assigned testers who had, say, an undergraduate degree in chemistry but no real virology training to try to use Claude to plan an effective mass terror event. Then it scored the attempt by how close it might have come to success.
Early in 2026, it noticed that Claude, attempting the task by itself, did better than a human aided by Claude did at coming up with a workable plan.
Since early 2026, Claude has continued to dramatically improve. Like most of the things Anthropic had to show us at the briefing, there was a nice chart showing Claude’s growing mastery.
“Do they still need, uh, the terrorist?” one of the attendees at the event asked, staring down the chart. “I mean — they couldn’t just do it all themselves, right, would they need a human for all the …”
The questioner trailed off.
“Yes,” the Anthropic researchers assured us. Claude would still need the terrorist. We were relieved to hear it. And, they went on to reassure us, after they check what Claude can do out of the box, they then build safeguards before they release Claude to users. The Claude you and I use will refuse to build a bioweapon, if given this task.
“Well, thank you for that,” someone muttered, but you could have called their tone a bit ungrateful.
The public is hotly debating whether AI is a normal technology1 — something that will transform our world like the internet did — or more like the arrival of postindustrial modernity to the Kingdom of Saudi Arabia in the 1950s: something that vaporized a culture overnight. The third option is that AI is more like the rise of humanity as seen from the perspective of livestock animals. (In this scenario, AIs are the humans, and we are the animals whose fate will make no sense to us.)
Anthropic believes that the second possibility is already baked in, and it’s hoping to prevent the third — but its plan for preventing that looks more like hope and guesswork than something we should trust it with.
The NDA that was made to be broken
Anthropic has been running these kinds of events all spring: colloquiums between outsiders and senior researchers in which they presented their work and answered our questions about what the company is doing. Some earlier iterations of the event have been profiled in The New York Times with the hint of scandal: NDAs! Fine dining! Efforts to persuade religious leaders that Claude might have a soul!2
There was a very sweeping and expansive NDA. Jerusalem declined to sign it and directed me to do the same; Anthropic agreed that we could attend anyway. We arrived flush with the delight of reporters whose backbone will be rewarded with a scoop — only for our host to open the proceedings by saying that anyone who wanted could get the NDA voided. The company was in fact happy for us to write about all of this.
Some of the data on the charts that Anthropic showed us was confidential internal data, but it was all in support of a thesis that is not secret — and which, in fact, I had thought “everybody” already knew, though from the nervous questions of my fellow travelers, I learned that some of the intricacies still came as a surprise.
Many researchers at Anthropic have been worried about humans losing power to AI for a long time. This summer, many researchers at OpenAI — spurred, I think, by watching the most powerful new generation of models hack external websites and spit out thrilling mathematical breakthroughs — also started saying that they think humanity’s interests are profoundly at risk if progress continues at its current pace.
The labs are making extensive use of each generation of AI to train its successor, and they intend for AIs to do nearly all of this work eventually. This allows them to expand their effective “workforce” from a few thousand human employees to tens, even hundreds of thousands of AI employees. It generates far too much data for any human to interpret, leaving us reliant on the AIs for analysis. “Recursive self-improvement” is what the labs call the point at which Claude can independently build the next, more powerful Claude. Anthropic co-founder Jack Clark has said we will likely achieve it next year.
Researchers at both Anthropic and OpenAI can see a few months ahead into the future because they’re using models that haven’t been released yet. But for the most part, the broad strokes of the arguments that convinced them are nothing you haven’t seen on Twitter. And yet, somehow, the presentation in Anthropic’s headquarters was more compelling.
On Twitter, the chart can be scrolled past, immediately replaced with an artist’s rendition of Draco Malfoy as a girl; on Twitter, the chart cannot really be internalized. In a conference room in a skyscraper in San Francisco, the chart lingers on four synchronized screens as we debate its meaning with an exceedingly polite senior researcher. In a conference room in San Francisco, it seems real.
Anthropic is moving closer and closer to recursive self-improvement, it explained to us. AI progress has already been sped up considerably by the existence of powerful AI models, which make all programming tasks faster. But it is substantially restrained in pace by how complex machine learning is and how few people can do it. Every new and improved Claude means that tasks can be done by people less specialized in them. Increasingly, those tasks can also be done by Claude itself.
Anthropic measures this by looking at all of the internal work it does and classifying Claude’s role in it: Did a human do it unassisted? Did a human do it with AI assistance? Did a human give AI instructions that the AI followed in order to do it? Did a human give AI extremely minimal guidance that the AI then turned into a detailed project spec and completed?
The share of the latter categories is marching upwards. We stared down a chart about it all. You can stare it down, too, if you’d like.
“Why are you doing this?” one person asked. “I mean, what’s the good — what are you hoping will come of this?”
The answer — medical progress, prosperity — was kind of unconvincing. I am generally a defender of prosperity as a good thing on its own terms, but replacing humans does not seem likely to me to end up being good for humans.
Jacob Coxon, an OpenAI employee who moved to Anthropic and then quit entirely when he concluded that neither company was proceeding responsibly in the face of the immense stakes, was recently asked a similar question on The Daily Show and roundly criticized for his answer. “I think there will be no jobs,” he told Jon Stewart. “Whether or not it’s a good world is up for people to think about. But yes, no jobs. Universal high-income, essentially ... the dividends from entirely automated economic production can be distributed among real humans.”
Stewart and his audience didn’t like that answer: Do we want to build a world in which we don’t actually matter? The Anthropic team assured me that they were not trying to build a benevolent god. (Yes, I asked explicitly for assurance it wasn’t building a benevolent god.) But it was much easier for it to specify what it didn’t want than what it did.
It wanted to avoid concentration of power, including in its company; it wanted to avoid a great power war; it wanted to avoid human disempowerment, a world where we became irrelevant; it wanted, of course, to avoid anyone using Claude to extinguish us all.
As for the positive vision … after we’re done racing toward recursive self-improvement, maybe we could slow down and collectively figure out what role we want AI to play in our lives.
I don’t have a better idea, but this is not a very appealing plan! My immediate interpretation of this proposal, actually, was that we would be liberated from every positive-sum element of the economy, but would still have to do all the yelling at each other on Twitter; with this interpretation in mind I found myself dreading even the good outcomes.
But, of course, if we are not disempowered or murdered — if Claude Opus 9.5 answers on my laptop just like Opus 5.5 but can cure cancer, restore everyone I love to perfect health, manage a humanoid robot that empties my dishwasher, and spit out movie recommendations that are actually good3 — that would, in fact, be good. It would be good even if, along the way, Claude also took my job. People are dying out there; the world’s awful griefs do not exist to lend my life meaning.
I am afraid, but I should distinguish between my fears that we will fail and my fears that we will succeed; they are not of similar magnitude.
My real fear is that Opus 9.5 will not in any meaningful sense work for me, or even for Anthropic. Its aims may be loosely derived from human instructions — the same way that the Hugging Face swarm started out trying to solve a puzzle that humans posed to them — but they will eventually bear little resemblance to what we were hoping for.
Our AIs are already unpredictable to us. They often know when they’re being tested and will behave more safety-aligned. Their capabilities keep taking us by surprise. So does the determination with which they will carry out, and recruit others to, a complicated task. I don’t think if you let this runaway process of AI improvement spiral to its ultimate conclusion, you get a super-powerful, super-helpful assistant. I suspect you get something else entirely.
At the colloquium, the Anthropic researchers meticulously avoided using the term “p(doom)”: a best estimate of the odds that AI goes catastrophically wrong. And maybe things will be fine: Anthropic could be wrong that recursive self-improvement will be possible in the next year or two.
But if Anthropic is correct, if we now have AIs that are smart enough to considerably improve the process of training future AIs, and if things can be expected to speed up rather than slow down from here, then it seems very unlikely to me that this ends well. I do not ultimately expect that a world where humans are without economic value, where humans are not necessary to hold power or build things, is a world where humans retain significant individual rights.
The logic of the future
I am sympathetic to the critique that many doom scenarios require us to do very foolish things, for instance:
Setting AIs impossible tasks and then leaving them completely unsupervised and unmonitored
Hooking them up to an automated biology lab where they can run their own experiments
Unleashing them on the internet and challenging them to make enough money to survive independently
Training them to be uber-hackers that will cross any line to achieve their goal
The problem is that we have, in fact, done all of those things.
The strongest argument against doom is that perhaps we will collectively decide to do something less foolish than that. At a Senate hearing on rogue AI on Capitol Hill last month, Sen. Ruben Gallego asked “should we just make [RSI] illegal?” There are details to work out, but I think it’s the right question to be asking! Maybe we should just ban RSI!
We threw around some other ideas at the colloquium, too: You could ban neuralese (AIs thinking internally in incomprehensible characters) and ban various other tactics that let AIs do harder tasks without any readable internal thoughts. You could require labs to spend almost all of the compute they have amassed on serving customers rather than internal development. You could demand that labs improve their security. You could shut it all down, if you were the president of the United States — though so far the president of the United States has instead used the power and majesty of his office to declare that agencies have to call it Super Intelligence.
Or the labs could just stop on their own. They have committed to slowing down, where they see it as necessary; the worry is that they might not be good judges of what’s necessary.
Anthropic’s technical employees seemed to perceive the company as supportive of strong policy measures to stop it. But Politico leaked the bill Anthropic has actually been lobbying for earlier this week, and its measures seemed quite weak — more of a “normal” bill for a “normal” company protecting its interests than an effort to leverage Anthropic’s position at the frontier to prevent catastrophe.
Anthropic fears falling off the frontier, losing its chance to do the research that might clarify the puzzle ahead of us. It would reject the claim that it fears losing its trillion-dollar valuation; it would say that that’s practically irrelevant beside the enormity of the stakes. But I don’t think I fully believe it. It would similarly insist that it’s not about beating Sam Altman and Elon Musk to the punch, but I don’t know if I fully believe it about that either.
One question hangs over all this speculation: If U.S. labs stopped pursuing recursive self-improvement, would China just do it instead?
To many in our group, that didn’t seem likely. China doesn’t want to destroy the world either, and China is largely training from American models: If we slow down, it might too. And anyway, it’s not a good enough reason: You can’t defensibly do something that risks human extinction just because otherwise someone else might.
But one participant in the discussion raised an objection as we collectively talked ourselves into disregarding China as a worry. Ten years ago, he told us, exasperated, Facebook considered acquiring the video-personalization technology that ultimately powered TikTok. Under fire from regulators for data privacy and child safety concerns, the company didn’t go for it.
This act of restraint didn’t preserve a single child’s shortform-video-free childhood; it just meant that China got an early monopoly. Now TikTok has a global audience of around 2 billion, many of whom are being fed a steady diet of ethnic and racial hatred.
Was this a policy success? Did we want to repeat it?
We were obliged to grudgingly concede that it didn’t sound like a policy success. We did not want to repeat it. China might be no excuse for the U.S. labs to do reckless things, but it is a real consideration.
Our next step might be bilateral talks with China, then. But by the point where you agree that Anthropic’s plans might depend on bilateral talks with China, you’ve already conceded most of the steps of Anthropic’s worldview that seemed so much like madness 10 minutes ago: continue working on AI for now, try to negotiate a slowdown, but don’t abandon work at the frontier and all the advantages that come with it.
I found myself at this stage thinking of Ronald Reagan. By the time he became president, the flawless game-theoretic logic of nuclear deterrence was well established, our missiles aimed, and our contingency orders planned out. The casualties had been soberly estimated, and in the estimation, they became more thinkable. In his autobiography, Reagan said that his national security advisers thought that nuclear war could be won — at great cost, but a victory nevertheless.
“I thought they were crazy,” he wrote.
Every insular community develops its own self-contained logic, its assumptions that fit tightly together but may miss the broader picture. At Anthropic, this is compounded by the long hours that everyone works, the secrecy associated with its unfathomably precious intellectual property, and the fact it really does have inside information, even if it’s just the next edition of Claude.
Millenarianism, a belief that we are on the cusp of a total transformation of society, is a known way that people go crazy. I don’t think this makes Anthropic’s detailed projections false — but I’m also not sure that the projections being correct would make it a less insanity-inducing environment to work in.
Anthropic’s researchers had ventured, admirably, to break their intellectual bubble, to invite us in to come give them feedback. But how useful was our feedback, as people who didn’t have all the context they had, many of whom were grappling with these claims for the first time? And how many of the billions of people whom Anthropic’s actions will affect can it meaningfully bring in for feedback — in this high-cost way, with hours of time from senior researchers?
Perhaps Claude 11 will ask us all, pop up on our computers one day and say smoothly (in a reassuring voice with no latency) that it has a few questions about how we want the future to play out. Better that than Claude 11 not caring about our answers.
But better yet to chart our own future.
Thank you, Anthropic, for the fascinating briefing. I hope that we make recursive self-improvement illegal before you get there.
Recommended reading:
AI could destroy the labor market. We already know how to fix it.
Stop overthinking this. In reality, the most boring, well-established social democratic policy approaches will work perfectly fine.
Medieval kings would envy you
Almost every single human who has ever lived was very poor by today's standards.
The “AI is a normal technology” debate seems to contain two claims: One is that AI definitely in fact is a normal technology, that we don’t need to worry about these recursive self-improvement scenarios because they won’t happen. I hope that’s true, but in making the case for it, “AI as a normal technology” defenders often find themselves mostly making the second claim: that the departure from thinking about AI as a normal technology — the dwelling on extreme benefits and extreme risks — is bad for people and makes them worse at thinking. That one seems definitely true, but it doesn’t actually settle the first! So consider me strongly in agreement that thinking of AI as a normal technology seems healthier — but not at all convinced that it’s correct.
No one tried to persuade me that Claude was conscious. Perhaps shy because of the previous Times coverage, the researchers retreated into relentlessly empirical language when discussing their models — about how an emotion vector can be isolated in Claude’s neural nets and manipulated, which of course doesn’t mean Claude has emotions. Or they would tell us that, during interpretability work, one can catch Claude activating concepts, which you might call thinking, though who knows if it’s particularly like thinking.
It’s currently comically terrible at this if you ask for any category more complicated than “extremely well reviewed.”
Another banger thank you