"Most people, when you describe what the labs are working on, recoil in horror." Do they though? I recall reading some excellent polling SOMEWHERE that while there is a noisy minority opposed, most people like the AI companies and their products: https://www.theargumentmag.com/p/four-things-i-changed-my-mind-on
I worry that the media industry is particularly threatened by AI and that leads to a bit of a "doom bubble" in reporting.
(Note that most people ARE quite NIMBY about big new data centers, but that is not the same as having issues with AI.)
It has only been recently that the media is opposing AI on the grounds that it and recursive self improvement are dangerous in and of themselves, not just for the “fake art”, “datacenters use water”, “student academic dishonesty” reasons
And that push has been largely the result of highly technical people, well involved in Silicon Valley, who have been thinking about this scenario and warning everyone about it for years upon years
I read that line as referring specifically to RSI.
I like the AI products I use and think that blocking data center construction is very dumb, but I’m decently horrified by the drive for RSI. I assume other people might have the same view.
RSI is something that the labs have always been open about, but is technical enough where the mass public isn't aware of it or the implications. I think those of us who have been paying a reasonable level of attention to AI development have known that automating AI research is the next most urgent goal of the labs. I hope this article does reach those who don't know about the concept of RSI yet, but what I was hoping for here is a more comprehensive view of what SHOULD be done.
Do we outright ban RSI? Force labs to submit each new model before deployment to be audited by some kind of government agency, existing or to be established? What about the legitimate national security concerns that China builds Skynet first?
Another angle is technical limitations. It always remains a possibility that AI progress will slow as we starve of new data to train on and limits of using synthetic data appear. We all know what would happen then - the massive capex spent would fail to generate the returns to pay off their financing and labs go kaput. More interesting is the Cowen critique that AI as currently constituted (bits not atoms) will hit frictions of uptake by human institutions. In business you'll slowly see small new firms built around AI workflows, some big firms adapt, and many fail to adapt, but that business cycle can take on the order of decades. Government could take even longer to adapt.
What are the implications of a machine god trapped in silicon? Robotics is, optimistically, at least a decade away from being at a place where LLMs we're in say, 2019. So we're not gonna have the machine god building robot factories so robots can build more robots and paperclip humans out of existence, at least not soon and not at a pace humans couldn't respond to. I think that framing, of a machine god that still is completely reliant on humans for real world action, could help guide what kind of regulations we need. Let the god spin it's wheels in data centers, perhaps, but once it asks us to build something we decide, and if we sense things are going weird we reboot the god?
Absolutely. Unfortunately given what we know of humanity “nobody does it” is not an option. Given the overall state of our technological development, and human propensity for some combination of greed/discovery/status/innovation (pick your own descriptor) it is highly unlikely that some actor somewhere won’t get there.
Very convenient thing to believe if you’re someone currently doing it
Not at all taking into account the fact that delaying it is at least something
Organizations like MIRI have well defined plans to implement an international pause treaty that tracks compute to prevent countries from continuing training. It could be tried, at least
"Most people, when you describe what the labs are working on, recoil in horror." Do they though? I recall reading some excellent polling SOMEWHERE that while there is a noisy minority opposed, most people like the AI companies and their products: https://www.theargumentmag.com/p/four-things-i-changed-my-mind-on
I worry that the media industry is particularly threatened by AI and that leads to a bit of a "doom bubble" in reporting.
(Note that most people ARE quite NIMBY about big new data centers, but that is not the same as having issues with AI.)
It has only been recently that the media is opposing AI on the grounds that it and recursive self improvement are dangerous in and of themselves, not just for the “fake art”, “datacenters use water”, “student academic dishonesty” reasons
And that push has been largely the result of highly technical people, well involved in Silicon Valley, who have been thinking about this scenario and warning everyone about it for years upon years
I read that line as referring specifically to RSI.
I like the AI products I use and think that blocking data center construction is very dumb, but I’m decently horrified by the drive for RSI. I assume other people might have the same view.
Anthropomorphism of AI is dangerous for the same reasons you cite. it's an abdication of responsibility.
RSI is something that the labs have always been open about, but is technical enough where the mass public isn't aware of it or the implications. I think those of us who have been paying a reasonable level of attention to AI development have known that automating AI research is the next most urgent goal of the labs. I hope this article does reach those who don't know about the concept of RSI yet, but what I was hoping for here is a more comprehensive view of what SHOULD be done.
Do we outright ban RSI? Force labs to submit each new model before deployment to be audited by some kind of government agency, existing or to be established? What about the legitimate national security concerns that China builds Skynet first?
Another angle is technical limitations. It always remains a possibility that AI progress will slow as we starve of new data to train on and limits of using synthetic data appear. We all know what would happen then - the massive capex spent would fail to generate the returns to pay off their financing and labs go kaput. More interesting is the Cowen critique that AI as currently constituted (bits not atoms) will hit frictions of uptake by human institutions. In business you'll slowly see small new firms built around AI workflows, some big firms adapt, and many fail to adapt, but that business cycle can take on the order of decades. Government could take even longer to adapt.
What are the implications of a machine god trapped in silicon? Robotics is, optimistically, at least a decade away from being at a place where LLMs we're in say, 2019. So we're not gonna have the machine god building robot factories so robots can build more robots and paperclip humans out of existence, at least not soon and not at a pace humans couldn't respond to. I think that framing, of a machine god that still is completely reliant on humans for real world action, could help guide what kind of regulations we need. Let the god spin it's wheels in data centers, perhaps, but once it asks us to build something we decide, and if we sense things are going weird we reboot the god?
“If we don’t do it someone you like a lot less will” is far from the craziest argument out there.
But you realize that the thing about a black box program is no matter who it came from, they can’t see inside and you can’t either
If nobody does it, no one will
Absolutely. Unfortunately given what we know of humanity “nobody does it” is not an option. Given the overall state of our technological development, and human propensity for some combination of greed/discovery/status/innovation (pick your own descriptor) it is highly unlikely that some actor somewhere won’t get there.
Very convenient thing to believe if you’re someone currently doing it
Not at all taking into account the fact that delaying it is at least something
Organizations like MIRI have well defined plans to implement an international pause treaty that tracks compute to prevent countries from continuing training. It could be tried, at least