I think part of the reason for discussing AI's danger in terms of complete human extinction, vs. something more nuanced like "85% of humanity will die before we cure the engineered plague / rebuild the grid / cease the global nuclear war" is that inevitably some people are going to imagine themselves at part of the lucky whatever%, and then we're off to fantasizing about repopulating the Earth, cool postapocalyptic fashion, and looting vacant mcmansions.
Plus, if we're talking about some sort of true superintelligence, actual extinction becomes plausible. Legions of robot warriors destroying the combined militaries of the world is currently science fiction nonsense; killer machines exterminating out the last million surviving humans in the Amazon and on isolated islands after successive virus dispersals wiped out everyone else is more realistic, conceptually.
Did apocalyptic language around nuclear weapons hinder nonproliferation efforts?
Somehow the "normal politics" around nuclear arms control managed to happen just fine. The same will be the case for AI. In the meantime please let's take seriously extinction risks from AI.
The existential risks aren’t so much one big event (like climate change) that we might be able to recover from. It’s more that a world run by a species other than us, on the basis of its interests, is likely to *keep* hitting us over and over again in different ways, one of which might finally take us all out. Think about the plight of various other species on an earth run by humans. Some of them, like coyotes and pigeons and raccoons, and now the parrots that are adapting to cities, are doing just fine. But many others are doing pretty badly. And out of the ones that are doing badly, an increasing number of them are getting some random combination of habitat loss and other changes to their environment that push them over the edge into extinction.
We could survive a single intentional mass nuclear exchange. But if, every year, the atmospheric chemistry changes, or the electricity available to humans goes away, or agricultural fields get replaced, or whatever, then our rebuilding efforts after one will get interrupted by the next, until one year the last of us are finally gone.
Fave part of the article was the reminder that the core imperative of human endeavor to resolve is that people disagree, and have to add that the main problem with p(doom) is how it tempts people to render particular deterministic cause and effect models probabilistic. This is useful in the sense that it informs the distribution of outcomes over the set of iterations, but that’s actively unhelpful for the N=1 choice point.
The analogy I like to use is in Magic: The Gathering. A matchup might be 65% overall, but that’s because you win every time you draw 1 of 2 cards in your opening hand. But if you’re looking at your opening hand and you understand the *probability* without the *mechanism*, the only thing the probability lets you do is make actively bad decisions: assuming a distribution is at play independent of its drivers. All you should be looking at - deterministically! - is whether your hand contains one of the two relevant cards.
So people who think that there's a chance that AI kills us all should just not say anything about that? Idk if that's true would seem kind of important to mention and discuss. And why is it fringe? Seems like it's becoming more mainstream as these systems have become more capable and are clearly displaying signals that they are difficult to control and understand. Especially now that the systems are approaching the ability to do recursive self improvement. If something really does have a chance to end human life how is it "fringe" to worry about it?
I've become skeptical/cynical about the recent calls from Open AI and Anthropic for a slowdown.
You have two companies spending huge amounts of money in amounts far out of scale relative to any profitability on products that a) seem not to be especially differentiated and b) face ruthless competition from not only each other but from many other sources.
Add to this that both companies want to go public to feed their cash needs.
If there's an existential risk it may be the survival of one or both of these companies in their present form with their current owners.
You can argue it's motivated reasoning, but it's not like tobacco executives lying about the effects of their product while knowing full well the truth.
I believe these guys 100% really do believe this stuff. You can doubt it, you can think they believe it because it matches incentives, but they aren't telling knowing lies.
Please read ai-2027.com
I think part of the reason for discussing AI's danger in terms of complete human extinction, vs. something more nuanced like "85% of humanity will die before we cure the engineered plague / rebuild the grid / cease the global nuclear war" is that inevitably some people are going to imagine themselves at part of the lucky whatever%, and then we're off to fantasizing about repopulating the Earth, cool postapocalyptic fashion, and looting vacant mcmansions.
Plus, if we're talking about some sort of true superintelligence, actual extinction becomes plausible. Legions of robot warriors destroying the combined militaries of the world is currently science fiction nonsense; killer machines exterminating out the last million surviving humans in the Amazon and on isolated islands after successive virus dispersals wiped out everyone else is more realistic, conceptually.
Did apocalyptic language around nuclear weapons hinder nonproliferation efforts?
Somehow the "normal politics" around nuclear arms control managed to happen just fine. The same will be the case for AI. In the meantime please let's take seriously extinction risks from AI.
The existential risks aren’t so much one big event (like climate change) that we might be able to recover from. It’s more that a world run by a species other than us, on the basis of its interests, is likely to *keep* hitting us over and over again in different ways, one of which might finally take us all out. Think about the plight of various other species on an earth run by humans. Some of them, like coyotes and pigeons and raccoons, and now the parrots that are adapting to cities, are doing just fine. But many others are doing pretty badly. And out of the ones that are doing badly, an increasing number of them are getting some random combination of habitat loss and other changes to their environment that push them over the edge into extinction.
We could survive a single intentional mass nuclear exchange. But if, every year, the atmospheric chemistry changes, or the electricity available to humans goes away, or agricultural fields get replaced, or whatever, then our rebuilding efforts after one will get interrupted by the next, until one year the last of us are finally gone.
Fave part of the article was the reminder that the core imperative of human endeavor to resolve is that people disagree, and have to add that the main problem with p(doom) is how it tempts people to render particular deterministic cause and effect models probabilistic. This is useful in the sense that it informs the distribution of outcomes over the set of iterations, but that’s actively unhelpful for the N=1 choice point.
The analogy I like to use is in Magic: The Gathering. A matchup might be 65% overall, but that’s because you win every time you draw 1 of 2 cards in your opening hand. But if you’re looking at your opening hand and you understand the *probability* without the *mechanism*, the only thing the probability lets you do is make actively bad decisions: assuming a distribution is at play independent of its drivers. All you should be looking at - deterministically! - is whether your hand contains one of the two relevant cards.
Also, Ratatouille is indeed the perfect film.
So people who think that there's a chance that AI kills us all should just not say anything about that? Idk if that's true would seem kind of important to mention and discuss. And why is it fringe? Seems like it's becoming more mainstream as these systems have become more capable and are clearly displaying signals that they are difficult to control and understand. Especially now that the systems are approaching the ability to do recursive self improvement. If something really does have a chance to end human life how is it "fringe" to worry about it?
I've become skeptical/cynical about the recent calls from Open AI and Anthropic for a slowdown.
You have two companies spending huge amounts of money in amounts far out of scale relative to any profitability on products that a) seem not to be especially differentiated and b) face ruthless competition from not only each other but from many other sources.
Add to this that both companies want to go public to feed their cash needs.
If there's an existential risk it may be the survival of one or both of these companies in their present form with their current owners.
You can argue it's motivated reasoning, but it's not like tobacco executives lying about the effects of their product while knowing full well the truth.
I believe these guys 100% really do believe this stuff. You can doubt it, you can think they believe it because it matches incentives, but they aren't telling knowing lies.