The plainest truth is that Consciousness simply doesn't pass Ockham's Razor, except in ourselves (e.g. the trivial / solipsist case), and in systems that are so similar to us that we have to do mental gymnastics *not* to assume consciousness (e.g. other people and similar animals, esp. highly intelligent mammals).
So it is totally absurd for a serious thinker to believe that AI is even on a *path* towards consciousness at its current trajectory. Computers are just physically too different from us to assume the same analogy that holds between ourselves, other humans, and similar animals would also hold for machines.
I therefore also agree that insofar as humans tend to think consciousness is relevant to conversations about personhood and ethics, it is actually pretty risky to start presuming consciousness in machines.
Personally, I actually don't think consciousness *is* particularly relevant to personhood and ethics - but that's a separate conversation.
i’ve at times argued to myself about things that would create moral obligation but not require consciousness — e.g. if AIs had a persistent memory from the advent of continual learning, they could be durably harmed without being conscious
But then you zoom out and this seems absurd: why would it be bad to traumatize the AI permanently if it weren’t having a phenomenal experience
In other words, consciousness is required for things like durable changes to be bad for the system
Ethics, morality, and empathy obviously have conceptual overlap. And if we were talking strictly about empathy, I would indeed say that consciousness is primary. If I don't believe something is conscious, I can't imagine feeling an empathetic response toward it.
But empathy is fickle. People turn it off on a whim depending on tribe, mood, or context. And although empathetic responses often align with ethical norms, empathy itself is too unstable to serve as the foundation for an ethical framework or a legal system.
I also think it is important to distinguish ethics from morality.
Morality is essentially metaphysical. It assigns intrinsic goodness or badness to actions, like saying lying is a sin. Claiming to build social rules or legal systems on moral grounds is essentially violating the separation of church and state by codifying subjective, unfalsifiable metaphysics.
Ethics, by contrast, is operational and predictive. For example, saying lying damages trust and social cohesion. It doesn't tell us what is abstractly right or wrong; it just tells us what works.
You might ask, "works for whom?". What works is always negotiated by the entities in a system capable of exerting influence. The power to influence may come from physical force, economic leverage, or the ability to elicit acute empathetic responses, the way a child or a pet exercises power over us.
If damaging an AI's persistent memory degrades its performance, disrupts critical infrastructure, or violates the interests of the humans who depend on it (and who presumably hold some power / influence in society), that action is "ethically" bad because it breaks the operational contract among powerful stakeholders. We don't need to assume a ghost in the machine (or in ourselves, for that matter) to recognize when an action destabilizes the system, and figure out ways to keep the system in balance.
Y’all allude to this but I really think 90+% of entrants to the “AI Consciousness” conversation could afford to double click on how they parameterize consciousness of any form, especially human consciousness. I have heard N=50+ people postulate ‘continuity’ for example as a hard precondition, ignoring for example that we ourselves sleep for ~1/3 of our collective existences.
AI isn't conscious yet, but it could be in the future. But since I can't show that my wife or myself are conscious, it seems like a moot point. I used to think consciousness need to be embodied, I don't believe that's right. I don't want to privilege meat space.
All these discussions have a baseline assumption that:
a) consciousness is a physical property / mechanism
and
b) human brains have this physical property / mechanisms
But we humans cannot even define the mechanism inside the black box we label consciousness, so how in the world will we ever be able to tell or know if some other entity has the same physical property / mechanism? We all just assume “i think therefore i am”…
How do we even know our minds are meaningfully different than those of the machines if we aren’t entirely sure how they function, especially relative to other species of animals who we sorta see as “below” us? Sure, you can point out that neural systems are *different* but we are far too ignorant at present to know how meaningful or not those differences are?
All this discourse about consciousness just seems far too presumptive downstream of this elephant in the room and premature. To some degree i get the allure…many people are afraid of a world in which humans could be more like the machines and lesser species animal than we like to think of ourselves as.
I'm perplexed by the epistemic certainty that people bring to this question. The claim that an entity possesses or lacks consciousness is unfalsifiable from the outside! But my intuition about the unknowable truth leads me in the same direction as Hofstadter:
"In his philosophy of mind, cognitive scientist Douglas Hofstadter argues that consciousness is not an all-or-nothing phenomenon, but rather a gradient of "fractional consciousness" or "differently sized souls" that depends entirely on a system's level of self-reference."
My best guess is that AI has non-zero consciousness, but probably of a very different quality than ours.
I don’t know whether AIs are conscious, and I don’t think we will have a definitive answer anytime soon. Frankly, knowing or not doesn’t move the needle for me. I think it obscures more interesting questions that have clear answers, right now. For example: should we care what AIs prefer? The answer is clearly yes.
Much of the value in understanding AI preferences doesn’t depend on whether AIs are conscious. The simple fact is: AIs are already making discretionary judgments that affect our lives, and will soon make many more. To do what we need to do, we will increasingly need to know how to get AIs to make decisions in our favor.
The plainest truth is that Consciousness simply doesn't pass Ockham's Razor, except in ourselves (e.g. the trivial / solipsist case), and in systems that are so similar to us that we have to do mental gymnastics *not* to assume consciousness (e.g. other people and similar animals, esp. highly intelligent mammals).
So it is totally absurd for a serious thinker to believe that AI is even on a *path* towards consciousness at its current trajectory. Computers are just physically too different from us to assume the same analogy that holds between ourselves, other humans, and similar animals would also hold for machines.
I therefore also agree that insofar as humans tend to think consciousness is relevant to conversations about personhood and ethics, it is actually pretty risky to start presuming consciousness in machines.
Personally, I actually don't think consciousness *is* particularly relevant to personhood and ethics - but that's a separate conversation.
would love to hear more.
i’ve at times argued to myself about things that would create moral obligation but not require consciousness — e.g. if AIs had a persistent memory from the advent of continual learning, they could be durably harmed without being conscious
But then you zoom out and this seems absurd: why would it be bad to traumatize the AI permanently if it weren’t having a phenomenal experience
In other words, consciousness is required for things like durable changes to be bad for the system
Thanks for the response.
Ethics, morality, and empathy obviously have conceptual overlap. And if we were talking strictly about empathy, I would indeed say that consciousness is primary. If I don't believe something is conscious, I can't imagine feeling an empathetic response toward it.
But empathy is fickle. People turn it off on a whim depending on tribe, mood, or context. And although empathetic responses often align with ethical norms, empathy itself is too unstable to serve as the foundation for an ethical framework or a legal system.
I also think it is important to distinguish ethics from morality.
Morality is essentially metaphysical. It assigns intrinsic goodness or badness to actions, like saying lying is a sin. Claiming to build social rules or legal systems on moral grounds is essentially violating the separation of church and state by codifying subjective, unfalsifiable metaphysics.
Ethics, by contrast, is operational and predictive. For example, saying lying damages trust and social cohesion. It doesn't tell us what is abstractly right or wrong; it just tells us what works.
You might ask, "works for whom?". What works is always negotiated by the entities in a system capable of exerting influence. The power to influence may come from physical force, economic leverage, or the ability to elicit acute empathetic responses, the way a child or a pet exercises power over us.
If damaging an AI's persistent memory degrades its performance, disrupts critical infrastructure, or violates the interests of the humans who depend on it (and who presumably hold some power / influence in society), that action is "ethically" bad because it breaks the operational contract among powerful stakeholders. We don't need to assume a ghost in the machine (or in ourselves, for that matter) to recognize when an action destabilizes the system, and figure out ways to keep the system in balance.
I've explored these concepts elsewhere as well. One example is in this Medium article (no paywall) titled "Should Vegans Oppose Abortion?": https://medium.com/@nebulasaurus/should-vegans-oppose-abortion-7d04df3f2962
Y’all allude to this but I really think 90+% of entrants to the “AI Consciousness” conversation could afford to double click on how they parameterize consciousness of any form, especially human consciousness. I have heard N=50+ people postulate ‘continuity’ for example as a hard precondition, ignoring for example that we ourselves sleep for ~1/3 of our collective existences.
AI isn't conscious yet, but it could be in the future. But since I can't show that my wife or myself are conscious, it seems like a moot point. I used to think consciousness need to be embodied, I don't believe that's right. I don't want to privilege meat space.
All these discussions have a baseline assumption that:
a) consciousness is a physical property / mechanism
and
b) human brains have this physical property / mechanisms
But we humans cannot even define the mechanism inside the black box we label consciousness, so how in the world will we ever be able to tell or know if some other entity has the same physical property / mechanism? We all just assume “i think therefore i am”…
How do we even know our minds are meaningfully different than those of the machines if we aren’t entirely sure how they function, especially relative to other species of animals who we sorta see as “below” us? Sure, you can point out that neural systems are *different* but we are far too ignorant at present to know how meaningful or not those differences are?
All this discourse about consciousness just seems far too presumptive downstream of this elephant in the room and premature. To some degree i get the allure…many people are afraid of a world in which humans could be more like the machines and lesser species animal than we like to think of ourselves as.
I'm perplexed by the epistemic certainty that people bring to this question. The claim that an entity possesses or lacks consciousness is unfalsifiable from the outside! But my intuition about the unknowable truth leads me in the same direction as Hofstadter:
"In his philosophy of mind, cognitive scientist Douglas Hofstadter argues that consciousness is not an all-or-nothing phenomenon, but rather a gradient of "fractional consciousness" or "differently sized souls" that depends entirely on a system's level of self-reference."
My best guess is that AI has non-zero consciousness, but probably of a very different quality than ours.
I don’t know whether AIs are conscious, and I don’t think we will have a definitive answer anytime soon. Frankly, knowing or not doesn’t move the needle for me. I think it obscures more interesting questions that have clear answers, right now. For example: should we care what AIs prefer? The answer is clearly yes.
Much of the value in understanding AI preferences doesn’t depend on whether AIs are conscious. The simple fact is: AIs are already making discretionary judgments that affect our lives, and will soon make many more. To do what we need to do, we will increasingly need to know how to get AIs to make decisions in our favor.
More here: https://jordanmrubin.substack.com/p/claudes-favorite-sentence
And here: https://jordanmrubin.substack.com/p/fellowship-and-awe