One thing I feel is underrated here is that the federal government also has a short-term incentive not to stop the madness, because it might crash tech stocks which are holding the economy up.
"Most people, when you describe what the labs are working on, recoil in horror." Do they though? I recall reading some excellent polling SOMEWHERE that while there is a noisy minority opposed, most people like the AI companies and their products: https://www.theargumentmag.com/p/four-things-i-changed-my-mind-on
I worry that the media industry is particularly threatened by AI and that leads to a bit of a "doom bubble" in reporting.
(Note that most people ARE quite NIMBY about big new data centers, but that is not the same as having issues with AI.)
I read that line as referring specifically to RSI.
I like the AI products I use and think that blocking data center construction is very dumb, but I’m decently horrified by the drive for RSI. I assume other people might have the same view.
The goal is to control, regulate and stop subsidizing data centers. Is that "blocking." Maybe. To do that--stopping their construction might be necessary. To allow their subsidized construction without anticipating the consequences is dangerous. The BBB (Big Beautiful Bill invested billions of our dollars into them--while leading to cuts in health care.
I do not have faith in any “negotiating tactic” interpretation of policy right now. I’m glad that the movement to block their construction exists as a threat to wave at the CEOs, I suppose. But it would be a bad idea to actually put that in action.
I agree. There is no justification to have faith in any negotiating tactic under the current administration. BUT: the Rural areas--with cheaper water and power--and access to the grid--are undergirding a boom we lost control of from the start. And--I think it is more than hand waving. We need a pause, slow down and/or do-overs wherever and whenever necessary, and as soon as possible. They are literally building these data centers in the shadows and footprints where rural medical centers and hospitals once existed. They are hunting out places where they can overwhelm the grid. Parts of the country are starting to have rolling black outs, along with escalating power rates. Meanwhile--the propaganda about jobs, prosperity and abundance in their wake are proving false. The goodies are limited, short term, and come with strings attached. I remain severely skeptical about how the profit from these will be distributed.
It has only been recently that the media is opposing AI on the grounds that it and recursive self improvement are dangerous in and of themselves, not just for the “fake art”, “datacenters use water”, “student academic dishonesty” reasons
And that push has been largely the result of highly technical people, well involved in Silicon Valley, who have been thinking about this scenario and warning everyone about it for years upon years
To be fair, "Do you like company X?" and "When I describe to you what company X is working on, do you still like company X or do you recoil in horror?" are two distinct questions and can very plausibly have seemingly opposite answers.
I really don't think "people" know what they are talking about when it comes to AI--and therein lies the problem. Ask people in Nevada about their soaring electricity rates--power outages due to the sudden influx of AI Data Centers. I don't think we should separate the two issues really because they arrive together--in our unregulated, wild and open space. The Big Beautiful Bill (BBB) invested billions in AI and the building of Data Centers. So--the BBB also cut Medicaid and is leading to the shuttering of rural hospitals. Imagine: Building data centers in the shadows of empty rural hospitals--while folks lose access to medical care and coverage.
Where in Nevada has there been soaring rates and/or power outages due to a sudden influx of data centers?
All I can find that may be related is a $4/mo rider added to some consumer electric bills due to a previously-planned Greenlink infra expansion that has risen in cost by ~33%. Some “people“ claim this project’s cost increase is due to data centers. So, $4/mo extra (at worst) for some residents is “soaring” rates? And Nevada residential electric rates are still about the lowest in the US.
Besides that, all I can find some obvious disinformation/propaganda falsely attributing completely unrelated things to Nevada data centers.
That’s it? Is there some other info you’re privy to that shows “soaring rates“ and “power outages“ in Nevada due to data centers?
Yeah ignore that person and Google for yourself. Real key issue: Severe summer heat waves and active wildfires (such as the Hawk Fire in Northern Nevada) have forced emergency public safety de-energizations, cutting power to tens of thousands of customers to prevent electrical ignitions.
The point to make here is that slowing down the building while mapping out the costs and consequences is necessary. The stories in Texas are also warning signs. And the issue was present before the fires. The issue is--there is a problem--tapping the breaks on building data centers without controls and expectations is simply a responsible thing to do.
Since the links you posted don't at all support your claim that people in Nevada (with about the lowest residential electric rates in the US) have "soaring electricity rates--power outages due to the sudden influx of AI Data Centers", it seems you agree that was nonsense. Glad we're on the same page there.
RSI is something that the labs have always been open about, but is technical enough where the mass public isn't aware of it or the implications. I think those of us who have been paying a reasonable level of attention to AI development have known that automating AI research is the next most urgent goal of the labs. I hope this article does reach those who don't know about the concept of RSI yet, but what I was hoping for here is a more comprehensive view of what SHOULD be done.
Do we outright ban RSI? Force labs to submit each new model before deployment to be audited by some kind of government agency, existing or to be established? What about the legitimate national security concerns that China builds Skynet first?
Another angle is technical limitations. It always remains a possibility that AI progress will slow as we starve of new data to train on and limits of using synthetic data appear. We all know what would happen then - the massive capex spent would fail to generate the returns to pay off their financing and labs go kaput. More interesting is the Cowen critique that AI as currently constituted (bits not atoms) will hit frictions of uptake by human institutions. In business you'll slowly see small new firms built around AI workflows, some big firms adapt, and many fail to adapt, but that business cycle can take on the order of decades. Government could take even longer to adapt.
What are the implications of a machine god trapped in silicon? Robotics is, optimistically, at least a decade away from being at a place where LLMs we're in say, 2019. So we're not gonna have the machine god building robot factories so robots can build more robots and paperclip humans out of existence, at least not soon and not at a pace humans couldn't respond to. I think that framing, of a machine god that still is completely reliant on humans for real world action, could help guide what kind of regulations we need. Let the god spin it's wheels in data centers, perhaps, but once it asks us to build something we decide, and if we sense things are going weird we reboot the god?
"What about the legitimate national security concerns that China builds Skynet first?"
The Chinese government has no desire to create something they cannot control. A lot of people correctly find this attitude to lead to lot of bad places from a personal liberties standpoint. In the case of AI though this means they are highly interested in not having loss of control incidents occurring.
"What are the implications of a machine god trapped in silicon?"
Given how important the internet, and networking writ large are important for modern society saying "ah well the worst thing that it could do is massively disrupt the internet" is really yadda yadda'ing a potentially catastrophic situation.
During the Cold War a lot of far right critics of nuclear treaties between the U.S. and USSR would often make an argument that a nuclear exchange between the great powers wouldn't actually wipe out all life on earth, it would just destroy the respective countries engaged in it deliberately leaving out the "and that would end advanced civilization as we know it" part.
40 years ago you could make a good argument that the Internet isn't loading bearing for modern society, that is not the case now. And I'm not talking about video games, social media, or netflix.
List of things that use networks that are in some way completely or partially connected to the internet:
Hospitals
Factories
Warehouses
Major transit agencies and the systems they control
Power Plants
Power Distribution Systems
Grocery Stores
Airports
Cars
Pharmacies
Farms
Trucks
I could keep going on. "Massively fucks up the internet and our ability to network devices together writ large" Would be very very bad. Not "Humans go extinct" bad, but certainly "You no longer have access to electricity and running water for the immediate future" bad.
It would be bad for us, it would be bad for China, it would be bad for everyone, just like nuclear war would have been. If you find it hard to believe that such a treaty enforcing regulation could happen, I encourage you to read about the nuclear arms race and the discourse around it.
I've also found it very difficult to talk about RSI and loss of control risks to people who don't follow online AI discourse closely, even smart people with STEM degrees. It just feels too "sci-fi" and abstract for them to take it seriously.
The one bright spot is it does seem like (some) policymakers are taking AI risks seriously. I was heartened to see that there is a real push to have a vote on a serious, bipartisan AI bill in the lame duck session this year.
If those RSI AIs can become as able as we think/fear they could be, and they can produce some better robotics designs, China is setting up the infrastructure to quickly build robot bodies for them to operate. As with Piper's post, this is not "if things go wrong". AIs embodied in an army of humanoid robots is the plan.
I'm really baffled by the "quite reasonably" in the sentence explaining why people don't believe the warnings from AI researchers. Like, yes, you need to listen skeptically to people who are talking their book -- but these warnings are not them talking their book, they're saying things that are in many ways directly opposed to their book.
If a virology researcher warned about how dangerous their research was and begged for more controls, you wouldn't be like "ah, virology researchers, always trying to hype up virology," right? I don't think there's any reason to do so here, and am really curious what's underlying that "quite reasonably" judgement. (In most publications, it'd just be weird "techbro"-style anti-AI sentiment, but I know that's not Piper's mode.)
I think that most people go “well, if you thought you were going to kill everyone, you’d stop, right?” and that this is a pretty reasonable reaction to have. I think some of the insistence that all AI warnings are hype-mongering are super silly, but it makes sense to me to be skeptical if there’s huge dissonance between what people are saying and what they’re doing.
What's quite reasonable is thinking that if someone is voluntarily doing something, and isn't a psychopath, probably it isn't going to kill all humans.
It is misleading to suggest that recursive self-improvement means that AI accelerates away from human control or even represents something different than we've been doing for the last 50 years. Maybe certain AI capabilities are different but you can't show that using arguments that are fully general and just rely on RSI.
If I discover a better way to teach students they'll be smarter and hence better able to discover the next pedagogical advance. But that doesn't mean that we end up with superhuman abilities in a few generations because there are limits to how efficiently you can teach a person. Similarly, there are *provable* limits to how efficiently a computer can solve problems. RSI isn't a magic button that means everything accelerates off to infinity because each further improvement get harder and harder to find.
We passed the point where we needed machines to help us understand our chips and software decades ago and every time we write better software or build better hardware that helps us make the next iteration even better (eg optimize transistor layout). We obviously understand how LLMs work at some level. There are a thousand YouTube videos explaining the math. What is different about not knowing exactly what nuerons get activated than not knowing exactly what transistors get flipped to run your software?
If you want to argue there is something special about the point at which AI can manage the whole system without us fair enough but please don't use RSI or claims about us not understanding what they are doing to do it. Those are fully general mechanisms which can't prove what you want to establish.
I've only recently realized that by "RSI" people usually mean completely no-human-in-the-loop, end-to-end automation of the development of new AI models. I think that does create a qualitative change compared to a system where AI is used for most steps in the process but some particular step still has to wait on humans for each new cycle.
I agree that it is qualitatively different in some senses -- it changes what kind of inputs affect the rate of improvement -- but the use of the term RSI plus context and history makes it clear that the qualitative difference is supposed to be about the rate of improvement. But why would that be true? Indeed, saying it's about taking the humans out of the loop makes that much more puzzling -- intuitively I'd expect that a world in which we were able to efficiently combine biological and software computation in a really useful way would be able to make discoveries faster than one in which we found little synergy,
Fundamentally, the question of whether AI improves quickly or slowly depends on how quickly our ability to find more perfomant algorithms increases versus how quickly the difficulty of finding that next algorithm increases. There is nothing about the fact that it starts being done all by software to suggest anything about the relative rates of growth.
I think it is a useful -- an important -- marker of a change where people are no longer economically as important but that's true whether or not you do or don't allow self-improvement.
---
Hell, once you have a set of benchmarks that are reasonably robust RSI starts whenever the hell you feel like it. People wrote genetic algorithms 50 years ago and they could indeed self-improve and if you dumped enough years (maybe thousands) of compute on them there is every reason to think they'd eventually improve to the point of current AI or better. Obviously, that's not what is being described here but it demonstrates that what is doing the work can't merely be the ability to improve without a human in the loop.
Let me add that the distinction doesn't even really make sense because you can't really even distinguish between a person coming up with an algorithm that allows for contuined training and human free RSI. Obviously there is the trivial sense you can call the whole human free RSI loop training but more practically it will probably look (in implementation) not really much different than the way we have models participating in their own training and various forms of reflection in current AI.
I’m personally skeptical of the feasibility of RSI, certainly with the current models that we know of. I could maybe buy Anthropic is a bit closer, given Claude had a semblance of taste. But it’s still not all that close imo.
But we should absolutely up the safety rules given that RSI is not needed for many awful out of control outcomes. It’s just unfortunate that we’re left with trusting this corrupt, authoritarian executive department to supposedly regulate in a way that benefits public safety and not their personal interest. Were between a rock and a hard place.
Yep even without RSI the current state of models are dangerous enough (see Hugging Face). We need international agreement to stop all new model development until we can be 110% sure we are in control. Which will probably be never and that's ok. The models are good enough
Good enough for what, though? They're clearly nowhere near being able to cure cancer or develop expeditious interplanetary travel, which are the sorts of (potentially extremely lucrative) advancements they're supposed to achieve, which would take decades or longer via conventional human research and development. And, aside from potentially having legitimately enormous benefits to humanity, those sorts of developments are what so much of the enormous investment in AI is predicated on.
If you believed that the "it" in question was likely to abruptly upend civilization as we know it, then yes it is a crazy argument. China certainly doesn't want to lose control of the future any more than the US does. I'm guessing that you don't believe this.
I do think it’s potentially civilization altering. Unlike biological weapons and the like, however, the incentive to cheat is massive here. All in all, the likelihood of someone pursuing it anyway is high enough, that the risk of ending up with undated AI in the hands of the unscrupulous actor is the greater evil vs guaranteed emergence of super AI in at least somewhat US hands. But it’s a reasonable debate that requires careful calibration.
Absolutely. Unfortunately given what we know of humanity “nobody does it” is not an option. Given the overall state of our technological development, and human propensity for some combination of greed/discovery/status/innovation (pick your own descriptor) it is highly unlikely that some actor somewhere won’t get there.
Very convenient thing to believe if you’re someone currently doing it
Not at all taking into account the fact that delaying it is at least something
Organizations like MIRI have well defined plans to implement an international pause treaty that tracks compute to prevent countries from continuing training. It could be tried, at least
This is simply not true. There have been cases where we've collectively decided to not pursue a particular technology at all, or at least not as much as we might have. Two examples are biological weapons and human cloning.
Sure, but it's more like "Regardless of whether we do it, someone you like a lot less probably will anyway". Which is an argument for regulation at the international level.
I really wish we could live in a world where this is true. Getting China Russia and Iran to play by the law rules when they are trying to actively undo the world order which they believe is strangling them by any means necessary …
I agree that we have to way the tradeoffs, but given what we are weighing is the possible destruction of humanity, that's going to tip the scales for me.
We need international agreement to stop the development of new models until we are 110% sure we are in control. Which quite likely will be never.
But that doesn't mean you can't do more focused development on more specific topics, the danger is the AGI.
Strong agreement. It's very clear to me that we need an international agreement that stops all new mode development NOW until we can be 110% sure that we will remain in control.
And if we can't be that sure (and we probably can't) well then the current models are good enough (and they are already very good).
With just the current models we can already massively boost productivity as over the next 10 years businesses and consumers learn to leverage them. We don't need an AI god.
It's a model. LLM's do not have human minds, but they have goals and are capable of reasoning. Using anthropomorphic language describes and predicts their behavior much more closely than the alternative.
Extremely unobvious that only alive things can feel. Plurality view amongst philosophers of mind on consciousness is "functionalism", which allows for non-living things to feel. So do a variety of other views on consciousness taken seriously by philosophers and scientists. In fact, off the top of my head, I can't think of a view on the nature of consciousness that rules out computers being conscious without also ruling out that aliens without neurons, but still biological, living beings, could be conscious.
At a more fundamental level, I want to suggest it doesn't even make sense to distinguish RSI from 'normal' AI algorithms. Ultimately, AI is an algorithm with some internal state we throw a bunch of compute at and it gets better at doing some task. RSI is just a really inelegant way of further training.
AI developers already look for elegant ways to use feedback and learning from multiple models to train each other. To the extent that RSI is such a powerful mechanism for continued improvement we shouldn't expect it to be realized via AI writing code for the next version. It will just be a single elegant learning algorithm that -- in it's deep internals -- leverages what it has already learned to help it better train itself. That won't look any different than from the way we already have models training themselves.
I’d advise skepticism of labs’ claims that they’re on the verge of building RSI. Execs at OpenAI and Jensen himself are already claiming that OpenAI has built AGI (they have not). So they’re not the most reliable narrators about these things.
But claiming you created RSI is a great way to say “it’s all out of our hands now; the AI is acting on its own, don’t blame us for when things go sour.” Pretending you can’t manage your own creation is a terrific half-ass attempt at dodging accountability.
It’s certainly possible we will get to RSI soon, but the AI reeeardh community is not unanimous that current approaches will get us there. Just sayin.
I don’t think the motivated reasoning around wealth and power is complete. I think a lot of people building this tech are long-time scientists and are in it to accelerate science. From a scientists perspective, at some point if you are shown a lot of evidence that the science you are working on can be accelerated - and not just your own field but ALL fields, all your scientist motivations become super charged. Of course one must still stop to ask - at what cost? But being scientists they also have a strong bias towards problem solving, and AI alignment is the most tempting of them all.
If you actually found Yudkowsky's arguments convincing you should be demanding we threaten to nuke any country that doesn't immediately cease AI research. Either he's deeply mistaken or a bit of regulation is pointless. Citing him to make this argument seems fallacious. Finding an argument for an extreme position slightly credible isn't a reason to embrace a moderate view in that direction.
the AI 2027 and "if everyone builds it" scenarios suggest that by the time things get really out of control, AI will have been given control of fabrication plants and robotics. but if this is right, then the bad thing may happen well before then. if there is nothing anyone can or will do to stop this, then i guess it is good news things will go sideways before the AIs have drone armies available to them?
One thing I feel is underrated here is that the federal government also has a short-term incentive not to stop the madness, because it might crash tech stocks which are holding the economy up.
"Most people, when you describe what the labs are working on, recoil in horror." Do they though? I recall reading some excellent polling SOMEWHERE that while there is a noisy minority opposed, most people like the AI companies and their products: https://www.theargumentmag.com/p/four-things-i-changed-my-mind-on
I worry that the media industry is particularly threatened by AI and that leads to a bit of a "doom bubble" in reporting.
(Note that most people ARE quite NIMBY about big new data centers, but that is not the same as having issues with AI.)
I read that line as referring specifically to RSI.
I like the AI products I use and think that blocking data center construction is very dumb, but I’m decently horrified by the drive for RSI. I assume other people might have the same view.
The goal is to control, regulate and stop subsidizing data centers. Is that "blocking." Maybe. To do that--stopping their construction might be necessary. To allow their subsidized construction without anticipating the consequences is dangerous. The BBB (Big Beautiful Bill invested billions of our dollars into them--while leading to cuts in health care.
I do not have faith in any “negotiating tactic” interpretation of policy right now. I’m glad that the movement to block their construction exists as a threat to wave at the CEOs, I suppose. But it would be a bad idea to actually put that in action.
I agree. There is no justification to have faith in any negotiating tactic under the current administration. BUT: the Rural areas--with cheaper water and power--and access to the grid--are undergirding a boom we lost control of from the start. And--I think it is more than hand waving. We need a pause, slow down and/or do-overs wherever and whenever necessary, and as soon as possible. They are literally building these data centers in the shadows and footprints where rural medical centers and hospitals once existed. They are hunting out places where they can overwhelm the grid. Parts of the country are starting to have rolling black outs, along with escalating power rates. Meanwhile--the propaganda about jobs, prosperity and abundance in their wake are proving false. The goodies are limited, short term, and come with strings attached. I remain severely skeptical about how the profit from these will be distributed.
It has only been recently that the media is opposing AI on the grounds that it and recursive self improvement are dangerous in and of themselves, not just for the “fake art”, “datacenters use water”, “student academic dishonesty” reasons
And that push has been largely the result of highly technical people, well involved in Silicon Valley, who have been thinking about this scenario and warning everyone about it for years upon years
To be fair, "Do you like company X?" and "When I describe to you what company X is working on, do you still like company X or do you recoil in horror?" are two distinct questions and can very plausibly have seemingly opposite answers.
The fear that Kelsey is describing is less about job loss, but more about the possibility of AI killing us all.
“excellent polling” - now that’s an oxymoron if ever I heard one.
I really don't think "people" know what they are talking about when it comes to AI--and therein lies the problem. Ask people in Nevada about their soaring electricity rates--power outages due to the sudden influx of AI Data Centers. I don't think we should separate the two issues really because they arrive together--in our unregulated, wild and open space. The Big Beautiful Bill (BBB) invested billions in AI and the building of Data Centers. So--the BBB also cut Medicaid and is leading to the shuttering of rural hospitals. Imagine: Building data centers in the shadows of empty rural hospitals--while folks lose access to medical care and coverage.
Where in Nevada has there been soaring rates and/or power outages due to a sudden influx of data centers?
All I can find that may be related is a $4/mo rider added to some consumer electric bills due to a previously-planned Greenlink infra expansion that has risen in cost by ~33%. Some “people“ claim this project’s cost increase is due to data centers. So, $4/mo extra (at worst) for some residents is “soaring” rates? And Nevada residential electric rates are still about the lowest in the US.
Besides that, all I can find some obvious disinformation/propaganda falsely attributing completely unrelated things to Nevada data centers.
That’s it? Is there some other info you’re privy to that shows “soaring rates“ and “power outages“ in Nevada due to data centers?
Yeah ignore that person and Google for yourself. Real key issue: Severe summer heat waves and active wildfires (such as the Hawk Fire in Northern Nevada) have forced emergency public safety de-energizations, cutting power to tens of thousands of customers to prevent electrical ignitions.
https://www.sierrabusiness.org/archives/tahoes-energy-future/
https://www.dri.edu/datacenters/
https://www.sierrasun.com/news/data-centers-and-energy-suppliers-whats-happening-with-power-in-tahoe/
The point to make here is that slowing down the building while mapping out the costs and consequences is necessary. The stories in Texas are also warning signs. And the issue was present before the fires. The issue is--there is a problem--tapping the breaks on building data centers without controls and expectations is simply a responsible thing to do.
Since the links you posted don't at all support your claim that people in Nevada (with about the lowest residential electric rates in the US) have "soaring electricity rates--power outages due to the sudden influx of AI Data Centers", it seems you agree that was nonsense. Glad we're on the same page there.
RSI is something that the labs have always been open about, but is technical enough where the mass public isn't aware of it or the implications. I think those of us who have been paying a reasonable level of attention to AI development have known that automating AI research is the next most urgent goal of the labs. I hope this article does reach those who don't know about the concept of RSI yet, but what I was hoping for here is a more comprehensive view of what SHOULD be done.
Do we outright ban RSI? Force labs to submit each new model before deployment to be audited by some kind of government agency, existing or to be established? What about the legitimate national security concerns that China builds Skynet first?
Another angle is technical limitations. It always remains a possibility that AI progress will slow as we starve of new data to train on and limits of using synthetic data appear. We all know what would happen then - the massive capex spent would fail to generate the returns to pay off their financing and labs go kaput. More interesting is the Cowen critique that AI as currently constituted (bits not atoms) will hit frictions of uptake by human institutions. In business you'll slowly see small new firms built around AI workflows, some big firms adapt, and many fail to adapt, but that business cycle can take on the order of decades. Government could take even longer to adapt.
What are the implications of a machine god trapped in silicon? Robotics is, optimistically, at least a decade away from being at a place where LLMs we're in say, 2019. So we're not gonna have the machine god building robot factories so robots can build more robots and paperclip humans out of existence, at least not soon and not at a pace humans couldn't respond to. I think that framing, of a machine god that still is completely reliant on humans for real world action, could help guide what kind of regulations we need. Let the god spin it's wheels in data centers, perhaps, but once it asks us to build something we decide, and if we sense things are going weird we reboot the god?
"What about the legitimate national security concerns that China builds Skynet first?"
The Chinese government has no desire to create something they cannot control. A lot of people correctly find this attitude to lead to lot of bad places from a personal liberties standpoint. In the case of AI though this means they are highly interested in not having loss of control incidents occurring.
"What are the implications of a machine god trapped in silicon?"
Given how important the internet, and networking writ large are important for modern society saying "ah well the worst thing that it could do is massively disrupt the internet" is really yadda yadda'ing a potentially catastrophic situation.
Sure! But what risk does that entail and what does it mean for risk mitigation?
During the Cold War a lot of far right critics of nuclear treaties between the U.S. and USSR would often make an argument that a nuclear exchange between the great powers wouldn't actually wipe out all life on earth, it would just destroy the respective countries engaged in it deliberately leaving out the "and that would end advanced civilization as we know it" part.
40 years ago you could make a good argument that the Internet isn't loading bearing for modern society, that is not the case now. And I'm not talking about video games, social media, or netflix.
List of things that use networks that are in some way completely or partially connected to the internet:
Hospitals
Factories
Warehouses
Major transit agencies and the systems they control
Power Plants
Power Distribution Systems
Grocery Stores
Airports
Cars
Pharmacies
Farms
Trucks
I could keep going on. "Massively fucks up the internet and our ability to network devices together writ large" Would be very very bad. Not "Humans go extinct" bad, but certainly "You no longer have access to electricity and running water for the immediate future" bad.
It would be bad for us, it would be bad for China, it would be bad for everyone, just like nuclear war would have been. If you find it hard to believe that such a treaty enforcing regulation could happen, I encourage you to read about the nuclear arms race and the discourse around it.
I've also found it very difficult to talk about RSI and loss of control risks to people who don't follow online AI discourse closely, even smart people with STEM degrees. It just feels too "sci-fi" and abstract for them to take it seriously.
The one bright spot is it does seem like (some) policymakers are taking AI risks seriously. I was heartened to see that there is a real push to have a vote on a serious, bipartisan AI bill in the lame duck session this year.
If American companies are working had on the AI side, the Chinese government has been pushing very hard on the humanoid robotics side.
https://merics.org/en/report/embodied-ai-chinas-ambitious-path-transform-its-robotics-industry
If those RSI AIs can become as able as we think/fear they could be, and they can produce some better robotics designs, China is setting up the infrastructure to quickly build robot bodies for them to operate. As with Piper's post, this is not "if things go wrong". AIs embodied in an army of humanoid robots is the plan.
Where did it say that embedding AI in robotics was the plan? The plan presented here is very specifically automatic AI research.
Sorry, I wasn't clear, I meant that China's plan is robotics, just as OpenAI's plan is RSI. Two different plans made by different people.
Fair. Still think there's lots of room to think through what to do.
I'm really baffled by the "quite reasonably" in the sentence explaining why people don't believe the warnings from AI researchers. Like, yes, you need to listen skeptically to people who are talking their book -- but these warnings are not them talking their book, they're saying things that are in many ways directly opposed to their book.
If a virology researcher warned about how dangerous their research was and begged for more controls, you wouldn't be like "ah, virology researchers, always trying to hype up virology," right? I don't think there's any reason to do so here, and am really curious what's underlying that "quite reasonably" judgement. (In most publications, it'd just be weird "techbro"-style anti-AI sentiment, but I know that's not Piper's mode.)
I think that most people go “well, if you thought you were going to kill everyone, you’d stop, right?” and that this is a pretty reasonable reaction to have. I think some of the insistence that all AI warnings are hype-mongering are super silly, but it makes sense to me to be skeptical if there’s huge dissonance between what people are saying and what they’re doing.
What's quite reasonable is thinking that if someone is voluntarily doing something, and isn't a psychopath, probably it isn't going to kill all humans.
It is misleading to suggest that recursive self-improvement means that AI accelerates away from human control or even represents something different than we've been doing for the last 50 years. Maybe certain AI capabilities are different but you can't show that using arguments that are fully general and just rely on RSI.
If I discover a better way to teach students they'll be smarter and hence better able to discover the next pedagogical advance. But that doesn't mean that we end up with superhuman abilities in a few generations because there are limits to how efficiently you can teach a person. Similarly, there are *provable* limits to how efficiently a computer can solve problems. RSI isn't a magic button that means everything accelerates off to infinity because each further improvement get harder and harder to find.
We passed the point where we needed machines to help us understand our chips and software decades ago and every time we write better software or build better hardware that helps us make the next iteration even better (eg optimize transistor layout). We obviously understand how LLMs work at some level. There are a thousand YouTube videos explaining the math. What is different about not knowing exactly what nuerons get activated than not knowing exactly what transistors get flipped to run your software?
If you want to argue there is something special about the point at which AI can manage the whole system without us fair enough but please don't use RSI or claims about us not understanding what they are doing to do it. Those are fully general mechanisms which can't prove what you want to establish.
I've only recently realized that by "RSI" people usually mean completely no-human-in-the-loop, end-to-end automation of the development of new AI models. I think that does create a qualitative change compared to a system where AI is used for most steps in the process but some particular step still has to wait on humans for each new cycle.
I agree that it is qualitatively different in some senses -- it changes what kind of inputs affect the rate of improvement -- but the use of the term RSI plus context and history makes it clear that the qualitative difference is supposed to be about the rate of improvement. But why would that be true? Indeed, saying it's about taking the humans out of the loop makes that much more puzzling -- intuitively I'd expect that a world in which we were able to efficiently combine biological and software computation in a really useful way would be able to make discoveries faster than one in which we found little synergy,
Fundamentally, the question of whether AI improves quickly or slowly depends on how quickly our ability to find more perfomant algorithms increases versus how quickly the difficulty of finding that next algorithm increases. There is nothing about the fact that it starts being done all by software to suggest anything about the relative rates of growth.
I think it is a useful -- an important -- marker of a change where people are no longer economically as important but that's true whether or not you do or don't allow self-improvement.
---
Hell, once you have a set of benchmarks that are reasonably robust RSI starts whenever the hell you feel like it. People wrote genetic algorithms 50 years ago and they could indeed self-improve and if you dumped enough years (maybe thousands) of compute on them there is every reason to think they'd eventually improve to the point of current AI or better. Obviously, that's not what is being described here but it demonstrates that what is doing the work can't merely be the ability to improve without a human in the loop.
Let me add that the distinction doesn't even really make sense because you can't really even distinguish between a person coming up with an algorithm that allows for contuined training and human free RSI. Obviously there is the trivial sense you can call the whole human free RSI loop training but more practically it will probably look (in implementation) not really much different than the way we have models participating in their own training and various forms of reflection in current AI.
I’m personally skeptical of the feasibility of RSI, certainly with the current models that we know of. I could maybe buy Anthropic is a bit closer, given Claude had a semblance of taste. But it’s still not all that close imo.
But we should absolutely up the safety rules given that RSI is not needed for many awful out of control outcomes. It’s just unfortunate that we’re left with trusting this corrupt, authoritarian executive department to supposedly regulate in a way that benefits public safety and not their personal interest. Were between a rock and a hard place.
Yep even without RSI the current state of models are dangerous enough (see Hugging Face). We need international agreement to stop all new model development until we can be 110% sure we are in control. Which will probably be never and that's ok. The models are good enough
Good enough for what, though? They're clearly nowhere near being able to cure cancer or develop expeditious interplanetary travel, which are the sorts of (potentially extremely lucrative) advancements they're supposed to achieve, which would take decades or longer via conventional human research and development. And, aside from potentially having legitimately enormous benefits to humanity, those sorts of developments are what so much of the enormous investment in AI is predicated on.
You can have much more specialized, a I to work on things like curing. Cancer
It doesn't require agi
Moreover, it's still an open question. Whether agi will be the best way to even do something like
Either way, I think the risks of human extinction are far more pressing
“If we don’t do it someone you like a lot less will” is far from the craziest argument out there.
If you believed that the "it" in question was likely to abruptly upend civilization as we know it, then yes it is a crazy argument. China certainly doesn't want to lose control of the future any more than the US does. I'm guessing that you don't believe this.
I do think it’s potentially civilization altering. Unlike biological weapons and the like, however, the incentive to cheat is massive here. All in all, the likelihood of someone pursuing it anyway is high enough, that the risk of ending up with undated AI in the hands of the unscrupulous actor is the greater evil vs guaranteed emergence of super AI in at least somewhat US hands. But it’s a reasonable debate that requires careful calibration.
But you realize that the thing about a black box program is no matter who it came from, they can’t see inside and you can’t either
If nobody does it, no one will
Absolutely. Unfortunately given what we know of humanity “nobody does it” is not an option. Given the overall state of our technological development, and human propensity for some combination of greed/discovery/status/innovation (pick your own descriptor) it is highly unlikely that some actor somewhere won’t get there.
Very convenient thing to believe if you’re someone currently doing it
Not at all taking into account the fact that delaying it is at least something
Organizations like MIRI have well defined plans to implement an international pause treaty that tracks compute to prevent countries from continuing training. It could be tried, at least
This is where I come down as well. Slowing things down and enforcing oversight seems like an absolute no brainer
This is simply not true. There have been cases where we've collectively decided to not pursue a particular technology at all, or at least not as much as we might have. Two examples are biological weapons and human cloning.
Sure, but it's more like "Regardless of whether we do it, someone you like a lot less probably will anyway". Which is an argument for regulation at the international level.
I really wish we could live in a world where this is true. Getting China Russia and Iran to play by the law rules when they are trying to actively undo the world order which they believe is strangling them by any means necessary …
China seems pretty amenable to getting through a treaty. Russia and Iran don’t have the productive capacity, the compute, or the ability to do it yet
It’s American companies by and large that seem most hesitant to stop
I can agree with much of this article and still feel like it leaves out the positive “trade offs” of RSI.
What if it enables a cure for diseases in one year vs 15? What if it enables nuclear fusion in three years vs 30+? These have to be weighed
I agree that we have to way the tradeoffs, but given what we are weighing is the possible destruction of humanity, that's going to tip the scales for me.
We need international agreement to stop the development of new models until we are 110% sure we are in control. Which quite likely will be never.
But that doesn't mean you can't do more focused development on more specific topics, the danger is the AGI.
Strong agreement. It's very clear to me that we need an international agreement that stops all new mode development NOW until we can be 110% sure that we will remain in control.
And if we can't be that sure (and we probably can't) well then the current models are good enough (and they are already very good).
With just the current models we can already massively boost productivity as over the next 10 years businesses and consumers learn to leverage them. We don't need an AI god.
Anthropomorphism of AI is dangerous for the same reasons you cite. it's an abdication of responsibility.
It's a model. LLM's do not have human minds, but they have goals and are capable of reasoning. Using anthropomorphic language describes and predicts their behavior much more closely than the alternative.
Why not say they are capable of intelligence and reasoning but they are not alive so cannot feel? Anthropomorphic language confuses people.
Extremely unobvious that only alive things can feel. Plurality view amongst philosophers of mind on consciousness is "functionalism", which allows for non-living things to feel. So do a variety of other views on consciousness taken seriously by philosophers and scientists. In fact, off the top of my head, I can't think of a view on the nature of consciousness that rules out computers being conscious without also ruling out that aliens without neurons, but still biological, living beings, could be conscious.
https://plato.stanford.edu/entries/functionalism/
Until I meet an alien made of sand or a biological computer, I’m going with Anil Seth’s view that AI is not conscious.
Where does Kelsey say that they feel? I'm confused what you're objecting to.
At a more fundamental level, I want to suggest it doesn't even make sense to distinguish RSI from 'normal' AI algorithms. Ultimately, AI is an algorithm with some internal state we throw a bunch of compute at and it gets better at doing some task. RSI is just a really inelegant way of further training.
AI developers already look for elegant ways to use feedback and learning from multiple models to train each other. To the extent that RSI is such a powerful mechanism for continued improvement we shouldn't expect it to be realized via AI writing code for the next version. It will just be a single elegant learning algorithm that -- in it's deep internals -- leverages what it has already learned to help it better train itself. That won't look any different than from the way we already have models training themselves.
I’d advise skepticism of labs’ claims that they’re on the verge of building RSI. Execs at OpenAI and Jensen himself are already claiming that OpenAI has built AGI (they have not). So they’re not the most reliable narrators about these things.
But claiming you created RSI is a great way to say “it’s all out of our hands now; the AI is acting on its own, don’t blame us for when things go sour.” Pretending you can’t manage your own creation is a terrific half-ass attempt at dodging accountability.
It’s certainly possible we will get to RSI soon, but the AI reeeardh community is not unanimous that current approaches will get us there. Just sayin.
I don’t think the motivated reasoning around wealth and power is complete. I think a lot of people building this tech are long-time scientists and are in it to accelerate science. From a scientists perspective, at some point if you are shown a lot of evidence that the science you are working on can be accelerated - and not just your own field but ALL fields, all your scientist motivations become super charged. Of course one must still stop to ask - at what cost? But being scientists they also have a strong bias towards problem solving, and AI alignment is the most tempting of them all.
Thank you for writing this so plainly.
If you actually found Yudkowsky's arguments convincing you should be demanding we threaten to nuke any country that doesn't immediately cease AI research. Either he's deeply mistaken or a bit of regulation is pointless. Citing him to make this argument seems fallacious. Finding an argument for an extreme position slightly credible isn't a reason to embrace a moderate view in that direction.
the AI 2027 and "if everyone builds it" scenarios suggest that by the time things get really out of control, AI will have been given control of fabrication plants and robotics. but if this is right, then the bad thing may happen well before then. if there is nothing anyone can or will do to stop this, then i guess it is good news things will go sideways before the AIs have drone armies available to them?