I think part of the reason for discussing AI's danger in terms of complete human extinction, vs. something more nuanced like "85% of humanity will die before we cure the engineered plague / rebuild the grid / cease the global nuclear war" is that inevitably some people are going to imagine themselves at part of the lucky whatever%, and then we're off to fantasizing about repopulating the Earth, cool postapocalyptic fashion, and looting vacant mcmansions.
Plus, if we're talking about some sort of true superintelligence, actual extinction becomes plausible. Legions of robot warriors destroying the combined militaries of the world is currently science fiction nonsense; killer machines exterminating out the last million surviving humans in the Amazon and on isolated islands after successive virus dispersals wiped out everyone else is more realistic, conceptually.
Did apocalyptic language around nuclear weapons hinder nonproliferation efforts?
Somehow the "normal politics" around nuclear arms control managed to happen just fine. The same will be the case for AI. In the meantime please let's take seriously extinction risks from AI.
I've become skeptical/cynical about the recent calls from Open AI and Anthropic for a slowdown.
You have two companies spending huge amounts of money in amounts far out of scale relative to any profitability on products that a) seem not to be especially differentiated and b) face ruthless competition from not only each other but from many other sources.
Add to this that both companies want to go public to feed their cash needs.
If there's an existential risk it may be the survival of one or both of these companies in their present form with their current owners.
You can argue it's motivated reasoning, but it's not like tobacco executives lying about the effects of their product while knowing full well the truth.
I believe these guys 100% really do believe this stuff. You can doubt it, you can think they believe it because it matches incentives, but they aren't telling knowing lies.
I think part of the reason for discussing AI's danger in terms of complete human extinction, vs. something more nuanced like "85% of humanity will die before we cure the engineered plague / rebuild the grid / cease the global nuclear war" is that inevitably some people are going to imagine themselves at part of the lucky whatever%, and then we're off to fantasizing about repopulating the Earth, cool postapocalyptic fashion, and looting vacant mcmansions.
Plus, if we're talking about some sort of true superintelligence, actual extinction becomes plausible. Legions of robot warriors destroying the combined militaries of the world is currently science fiction nonsense; killer machines exterminating out the last million surviving humans in the Amazon and on isolated islands after successive virus dispersals wiped out everyone else is more realistic, conceptually.
Please read ai-2027.com
Did apocalyptic language around nuclear weapons hinder nonproliferation efforts?
Somehow the "normal politics" around nuclear arms control managed to happen just fine. The same will be the case for AI. In the meantime please let's take seriously extinction risks from AI.
I've become skeptical/cynical about the recent calls from Open AI and Anthropic for a slowdown.
You have two companies spending huge amounts of money in amounts far out of scale relative to any profitability on products that a) seem not to be especially differentiated and b) face ruthless competition from not only each other but from many other sources.
Add to this that both companies want to go public to feed their cash needs.
If there's an existential risk it may be the survival of one or both of these companies in their present form with their current owners.
You can argue it's motivated reasoning, but it's not like tobacco executives lying about the effects of their product while knowing full well the truth.
I believe these guys 100% really do believe this stuff. You can doubt it, you can think they believe it because it matches incentives, but they aren't telling knowing lies.