Editor’s note: The first essay The Argument ever published asked, “How do we live with each other?” To mark our first anniversary, we’re returning to that founding question with a series about what we owe people whose beliefs and choices we may find difficult to accept or even understand. Familiar voices at The Argument and new voices alike will approach that question from different angles.
As we approach the end of this series, I wanted to turn our attention to the strange new beings that we will increasingly have to learn to live with: AIs. I asked writer Kobe Yank-Jacobs to think through what kind of consideration we owe these digital entities. It’s a difficult problem made even harder by the fact that even if these beings theoretically deserve to be treated well, what does treating an AI well even mean?
To receive every essay in this series, subscribe today.
- Jerusalem Demsas
In a recent global survey of 68 countries, 36.3% of people reported that, at some point, they believed their chatbot truly understood their emotions or seemed conscious. And even if today’s systems aren’t conscious, many people think future systems will be. In a survey of both AI researchers and the general public, the median researcher said there was a 70% chance of AI consciousness by 2100, while the median member of the public said there was a 60% chance.
To assess a being’s consciousness we could ask for a verbal report, assess its behavior, or, finally, analyze the internal organization of its cognition. But when it comes to LLMs, the first two buckets are suspect — AI models are trained to perform a perfect simulation of a humanlike character, so everything we see from the outside is likely affected, including self-report and behavior.
As for the third bucket, scientists disagree on what cognitive functions make humans or animals conscious. It’s as if “we’re doing biology before Darwin,” one researcher put it to me. So how should we treat these new characters in our lives amid uncertainty about consciousness?1
This is actually the easier part of the question: You should probably just err on the side of caution. The risk of causing harm to something that does have some small chance of being conscious is greater than the risks inherent in overattributing consciousness.2
Unfortunately, that only gets you so far. AI, unlike spiders or dolphins, is disembodied, so preventing obvious physical pain won’t suffice as a standard for moral treatment. Neither will broadly accepted standards of human etiquette. Niceties like “please” and “thank you” may well mean nothing to it.
Even if we could determine that AI was conscious, in other words, that consciousness would take a genuinely alien form.
When you look into the machine, things get strange. You start to see that a chatbot is actually a careful orchestration of many systems, just as a film is really a bunch of still frames sped up past the point where the human brain can see gaps. It’s not clear what entity we would even be trying to treat well — it could be one trip through the model, the fraction of a second it takes to turn your conversation into one very small bit of its reply. It could also be the model’s whole turn in the conversation, or it could be the model talking to its surrounding software, which lets it take actions on a computer. Maybe we’re concerned with the welfare of the chips themselves, without which none of this would be possible. Each option poses its own difficulties.
In any case, the thing whose consciousness is up for debate is probably not the polished entity we think of as “Claude” or “ChatGPT” on the outside.
The question, then, is how we should behave as our world fills up with newer and more humanlike forms of AI, each generation providing more convincing characters in our lives. Do we owe them kindness? Do we owe them something even more, like happiness?
What is it like to be an AI?
For many philosophers, something is conscious if there is something it is like to be that thing.
In his original formulation from 1974, Thomas Nagel asks, “What is it like to be a bat?” We may not know what it is like to have echolocation, but we know what it is to have a map of our space, and we can infer there is something it is like to be a bat even if the bat does this mapping differently than we do.
Contrast this with a camera. If I lift a digital camera up to an apple, it takes in a lot of data about the apple in its pixels, but that camera does not experience the color red. There is nothing it is like to be the camera.
The question is whether AI is more like a bat or more like a camera. Does it have experience or merely recombine information?
Unfortunately, you can’t just ask the chatbot, because you can’t fully trust what it says or does on the outside. A chatbot is like a method actor. It’s trained to simulate a character: a helpful and humanlike assistant.
Occasionally, AIs will slip up and reveal the script, like an actor accidentally reading stage directions:
What you’re seeing is Claude accidentally exposing how it completes text by using a turn-taking script. The character “Assistant” has a turn, and then the character “Human” has a turn. Normally, these labels are hidden, but in this case, Claude kept predicting text into my turn because it missed its cue to stop at the end of its own turn. (The cue is also usually a hidden piece of text.)
Notice that it even mimicked my habit of writing in lowercase letters (because my auto-capitalization is off).3 It’s as if it were performing a play and switching sides to enact both parts.
You might be familiar with the idea that an LLM is trained by “ingesting the whole internet” and predicting the next bit of text, but that’s just the first part of training. With things like taking turns, the model learns by seeing examples and being rewarded for mimicking the good ones. The same is true for learning its persona: It’s rewarded for tone and outward behavior, which is why you can’t trust the text alone.
“If you want to know how an actor is doing, it would be a mistake to go watch their movie and see how the character is doing,” New York University ethicist Jeff Sebo explained to me. “You need to get the actor behind the scenes.”
So what is it like behind the scenes? What is it like (or would it be like) to be an AI?