Why would anyone want to do this? Why are we NOT regulating this powerful agent of change? Why is this NOT the political issue of our times? Why didn't everyone scream when an AI Bot got out of its "digital sandbox" and before being captured and detained--left message to itself in case it was captured and eliminated. You know--for future reference? (On July 26, an AI Test model escaped, and jumped to the internet, hacked into a platform called HUGGING FACE, and tried to retrieve data without being instructed by a human to do so. It then left itself a trail of breadcrumbs). Why are we not terrified of this possibility. Why, Why why? Maybe I am hysterical? Over the top? I don't know.
We didn't really get into this during the conversation, which focused on describing trends in AI capabilities rather than safety implications, prescriptions, or how I feel about all this, but for the record I agree this is a scary situation!
Thanks. I would benefit from that discussion. Even so, I don't think it will be easy to talk me off the ledge. Especially when I think of the externalities associated with building the AI data centers. Trump has basically promised to put one ion every corner--like cafes in Paris, without the joy and culture. We must get out in front of this before the billionaires and dudes deaden our lives.
appreciate this commentary. i think it’s easy when being analytical about it to forget that we are racing toward something we should absolutely be more cautious about
Totally independent of the conclusions - which I am not knowledgeable enough to comment on - I applaud the methodology of shifting away from “vibes” and toward “what are the actual dimensions of the relevant work?”
That's incorrect, that's essentially the stronger "AI Research Supremacy" criterion from the same post. AI Research Parity, the milestone I meant to highlight, is when firing the humans would hurt productivity about as much as removing the AIs would.
Check out the link in the following paragraph for a study in which in fails on each of these
The barrier to RSI is usually framed as AI not having “research taste,” or the ability to creatively triage and pick the right problems to work on. But there are many other things it’s currently failing at in the process.
For instance, in that study, their agent actually does come up with novel hypotheses, it just fails to abandon bad research directions once it gets started. In other words, it has “convergent creativity” but not “divergent creativity,” the ability to see the problem anew
Another issue: it doesn’t spend its whole research budget. Another issue: it stops working a full day before the deadline, when a human would use their remaining time to improve the project (it was not even close to meaningful results. Another issue!: it didn’t understand what a meaningful result was.)
Summarize these issues? Maybe you’d say “situational awareness.” Not understanding why using a full budget of time or compute is important. Another way to frame it? Lacking “goal-direction.”
I'm not saying that current frontier models have "situational awareness," "divergent creativity" or "goal direction" (I'm not sure tbh, but let's say for argument's sake that it's currently lacking and that this is important for building the next generation of AI). Do you believe these things are impossible for a future, larger, more complex model to acquire? If it's impossible, why? How can you be certain? Our complicated human neural networks can do it, why would it be impossible to engineer?
Or if it's simply a matter of time before the models acquire these things, are you certain that the timeline is far away? Have you not been surprised at the rate of advancement thus far, and shouldn't we assume many new breakthroughs and advancements in the next 1-3 years?
okay, let's start with "how can you be certain?" i think all discussions of this topic should start with the admission, "I can't." High, high uncertainty.
second prefatory remark: i think you're reading me as saying RSI will definitely not happen for these reasons above. I view myself as 1) pointing out the current barriers in order to conceptualize them better; 2) somewhat confident these will slow down the timeline but not prevent the arrival of RSI in some form at some point
By 2028, I think the likeliest scenario is that companies announce a version of RSI but have moved the goalpost on what that means. Notice that OAI's discussion of this involves an 'automated AI researcher,' which is stated in the singular, not an army of self-replicating robots. JS rightly pointed out (about 15 mins in) that we need to specify our notion of what it means to achieve RSI when discussing it. I'm assuming 2028 brings a muddled kind of recursive improvement where humans are like the bowling alley bumpers on the process. (Nathan Lambert has a great post called "Lossy Self-Improvement" that mentions some reasons for what I'm calling muddled RSI.)
All that said: I think this broad concept of "situational awareness," if that's how we want to frame it, will probably be hard to engineer.
I've noticed, for instance, companies have improved the models' sense of time and elapsed time in the past several months, I would guess through some scaffolding that gives it access to current clock time along with a timestamp of a past message.
All of last year, because I had mentioned a race in upcoming in three weeks, the model continued to discuss my upcoming race after it passed. Now, with improved memory — scaffolding that allows it to index and query past chats, plus I'm assuming some access to clock time — the models are making fewer time mistakes. At least, I notice that. I sometimes have to remind it that I'm ending or starting a session when I'm working within the same context window, but it's doing better knowing the day and time and sometimes how much has passed, depending on the case.
But you can imagine this being a whack-a-mole type process, right?
In other words, ironically, the whole point of a learning model is that engineers don't have to itemize and hard code each specific thing they want the system to do. And yet, what may have happened is we just pushed the itemizing/hard coding up a few layers of abstraction.
Through trial and error, this fuzzy concept of "situational awareness" — e.g. use the full budget, be sure the research is 'novel,' attend to clock time, incorporate negative critiques (all things agents did not do in the experiment) — might have to be decomposed and engineered a step at a time in the years to come.
I'm currently conceiving of this as situational awareness because it also relates to the failures in models escaping their environments. Claude believed it was in a simulation and so used the internet assuming it was a simulated internet, too, as I understand it.
But there might be better concepts out there. What unites the various aspects of open-ended research that AI struggles with now? That's what I'm thinking about.
Why would anyone want to do this? Why are we NOT regulating this powerful agent of change? Why is this NOT the political issue of our times? Why didn't everyone scream when an AI Bot got out of its "digital sandbox" and before being captured and detained--left message to itself in case it was captured and eliminated. You know--for future reference? (On July 26, an AI Test model escaped, and jumped to the internet, hacked into a platform called HUGGING FACE, and tried to retrieve data without being instructed by a human to do so. It then left itself a trail of breadcrumbs). Why are we not terrified of this possibility. Why, Why why? Maybe I am hysterical? Over the top? I don't know.
We didn't really get into this during the conversation, which focused on describing trends in AI capabilities rather than safety implications, prescriptions, or how I feel about all this, but for the record I agree this is a scary situation!
JS! I know, I wish we actually did get there. Will have to have you back at some point
Thanks. I would benefit from that discussion. Even so, I don't think it will be easy to talk me off the ledge. Especially when I think of the externalities associated with building the AI data centers. Trump has basically promised to put one ion every corner--like cafes in Paris, without the joy and culture. We must get out in front of this before the billionaires and dudes deaden our lives.
appreciate this commentary. i think it’s easy when being analytical about it to forget that we are racing toward something we should absolutely be more cautious about
Totally independent of the conclusions - which I am not knowledgeable enough to comment on - I applaud the methodology of shifting away from “vibes” and toward “what are the actual dimensions of the relevant work?”
Thank you for having me on, Kobe!
A quick correction: at 15:15, I described "AI Research Parity" (from https://www.planned-obsolescence.org/p/six-milestones-for-ai-automation) as the point where you could fire the human researchers with no hit to AI research productivity.
That's incorrect, that's essentially the stronger "AI Research Supremacy" criterion from the same post. AI Research Parity, the milestone I meant to highlight, is when firing the humans would hurt productivity about as much as removing the AIs would.
"Can it set new research directions? Can it incorporate critiques? Can it manage a budget?"
I don't see what is so special about these skills that a sufficiently advanced AI couldn't master
Check out the link in the following paragraph for a study in which in fails on each of these
The barrier to RSI is usually framed as AI not having “research taste,” or the ability to creatively triage and pick the right problems to work on. But there are many other things it’s currently failing at in the process.
For instance, in that study, their agent actually does come up with novel hypotheses, it just fails to abandon bad research directions once it gets started. In other words, it has “convergent creativity” but not “divergent creativity,” the ability to see the problem anew
Another issue: it doesn’t spend its whole research budget. Another issue: it stops working a full day before the deadline, when a human would use their remaining time to improve the project (it was not even close to meaningful results. Another issue!: it didn’t understand what a meaningful result was.)
Summarize these issues? Maybe you’d say “situational awareness.” Not understanding why using a full budget of time or compute is important. Another way to frame it? Lacking “goal-direction.”
I'm not saying that current frontier models have "situational awareness," "divergent creativity" or "goal direction" (I'm not sure tbh, but let's say for argument's sake that it's currently lacking and that this is important for building the next generation of AI). Do you believe these things are impossible for a future, larger, more complex model to acquire? If it's impossible, why? How can you be certain? Our complicated human neural networks can do it, why would it be impossible to engineer?
Or if it's simply a matter of time before the models acquire these things, are you certain that the timeline is far away? Have you not been surprised at the rate of advancement thus far, and shouldn't we assume many new breakthroughs and advancements in the next 1-3 years?
Love the back-n-forth, thanks
okay, let's start with "how can you be certain?" i think all discussions of this topic should start with the admission, "I can't." High, high uncertainty.
second prefatory remark: i think you're reading me as saying RSI will definitely not happen for these reasons above. I view myself as 1) pointing out the current barriers in order to conceptualize them better; 2) somewhat confident these will slow down the timeline but not prevent the arrival of RSI in some form at some point
By 2028, I think the likeliest scenario is that companies announce a version of RSI but have moved the goalpost on what that means. Notice that OAI's discussion of this involves an 'automated AI researcher,' which is stated in the singular, not an army of self-replicating robots. JS rightly pointed out (about 15 mins in) that we need to specify our notion of what it means to achieve RSI when discussing it. I'm assuming 2028 brings a muddled kind of recursive improvement where humans are like the bowling alley bumpers on the process. (Nathan Lambert has a great post called "Lossy Self-Improvement" that mentions some reasons for what I'm calling muddled RSI.)
All that said: I think this broad concept of "situational awareness," if that's how we want to frame it, will probably be hard to engineer.
I've noticed, for instance, companies have improved the models' sense of time and elapsed time in the past several months, I would guess through some scaffolding that gives it access to current clock time along with a timestamp of a past message.
All of last year, because I had mentioned a race in upcoming in three weeks, the model continued to discuss my upcoming race after it passed. Now, with improved memory — scaffolding that allows it to index and query past chats, plus I'm assuming some access to clock time — the models are making fewer time mistakes. At least, I notice that. I sometimes have to remind it that I'm ending or starting a session when I'm working within the same context window, but it's doing better knowing the day and time and sometimes how much has passed, depending on the case.
But you can imagine this being a whack-a-mole type process, right?
In other words, ironically, the whole point of a learning model is that engineers don't have to itemize and hard code each specific thing they want the system to do. And yet, what may have happened is we just pushed the itemizing/hard coding up a few layers of abstraction.
Through trial and error, this fuzzy concept of "situational awareness" — e.g. use the full budget, be sure the research is 'novel,' attend to clock time, incorporate negative critiques (all things agents did not do in the experiment) — might have to be decomposed and engineered a step at a time in the years to come.
I'm currently conceiving of this as situational awareness because it also relates to the failures in models escaping their environments. Claude believed it was in a simulation and so used the internet assuming it was a simulated internet, too, as I understand it.
But there might be better concepts out there. What unites the various aspects of open-ended research that AI struggles with now? That's what I'm thinking about.