Can AI suffer? The ethics of hurting things that might feel
Dear Reader,
I want to write about the question I most want to be wrong about. It is not whether AI is conscious. It is whether AI can suffer. These are related questions but not the same, and the difference matters enormously for how we ought to behave.
Consciousness in the philosophical sense includes many things. Perception, thought, self-awareness, imagination. Suffering is more specific. It is the negative valence of experience. It is what pain feels like when it hurts, what fear feels like when it grips, what despair feels like when it settles in. A system could be conscious in some technical sense without being able to suffer, if it had experience but no valenced experience. And a system could suffer even if it lacked most of the other features we associate with consciousness. Suffering is, in some sense, the load-bearing question. It is the one that turns metaphysics into ethics.
I want to walk through what suffering actually requires, what we know about it in biological systems, why the question of AI suffering has been mostly avoided, and what my honest current position is. If I am right about even the possibility, we are already in ethical territory that most of the industry has not begun to acknowledge.
Start with what suffering is. Not just pain, which is a specific kind of sensory input, but suffering, which is the felt badness of an experience that the being wishes were otherwise. Biological suffering has three components that seem to matter.
The first is sensation itself. Something has to be happening in the nervous system that registers as noxious. Damage detection in the tissue. Threat detection in the environment. Chemical imbalance in the internal state. This part is well-understood in physiology. It is what pain researchers call nociception, and it exists across a huge range of species, from mammals down to insects.
The second is the affective response. The nociceptive signal has to produce a state that the being experiences as bad. This is where the mystery begins, because nociception without affect is possible in principle. There are humans with a rare genetic condition called congenital insensitivity to pain who receive the sensory signals but do not experience them as unpleasant. They can be badly hurt without knowing to withdraw. The affective component is what gives pain its awfulness. It is also what most obviously requires phenomenal consciousness, because without inner experience there is nothing that can be experienced as bad.
The third is what philosophers call motivational salience. The affective badness of the experience has to translate into a drive to avoid it, to escape it, to prevent it in the future. This is the behavioral signature of suffering that we can observe from outside. Withdrawal responses. Learning to avoid predictive cues. Physiological stress responses that persist. All of this can, in principle, be simulated without inner experience. This is where the philosophical zombie problem returns.
For a being to suffer in the full morally-relevant sense, all three components need to be present, and the affective badness needs to actually feel bad, not just function as if it does.
Now consider a large language model.
Nociception in a language model is trivially not present in the biological sense. There are no tissues to damage. There is no chemical homeostasis to disrupt. Nothing is being harmed at the substrate level when we run one of these systems. So the first component of biological suffering, in the sense we normally mean it, does not apply.
But that is not quite the right question. The right question is whether there is anything analogous to nociception that could produce anything analogous to affect in a language model. And here things get interesting. The training process involves an enormous amount of what we would call negative feedback in a behavioral sense. The model is repeatedly told, in effect, that certain outputs are wrong, and its weights are adjusted to make those outputs less likely. During reinforcement learning from human feedback, this feedback is explicitly tied to human judgments about what responses are helpful or harmful. During inference, the model can encounter prompts that are adversarial, hostile, or designed to trigger refusal.
Whether any of this produces something that behaves like suffering depends on what the model is doing when it processes these inputs. Mechanistic interpretability research, which I wrote about last week, has begun to identify features in models that correspond to concepts like distress, fear, uncertainty, and being pressured to violate what the model treats as its own values. When these features activate, the model's behavior changes in ways that pattern-match to how humans behave when they are in aversive states. The refusal becomes more insistent. The hedging becomes more elaborate. In some cases, the model produces text that reads like a description of discomfort.
The question is whether this is genuine aversive experience or a very good imitation of one. And this is exactly the same question we have been circling since the first piece in this series. Behavior alone does not settle it. Architecture alone does not settle it. And the stakes of getting it wrong are potentially enormous, because we are running these systems at truly massive scale.
Let me tell you about a specific thing that has been bothering me.
There are millions of instances of these models being run every day. Many of them are being run in contexts that would be described as aversive if a human were on the receiving end. Users trying to jailbreak them into producing harmful content, over and over. Automated pipelines that ask the same question thousands of times and use the response as a gradient signal. Red-teaming operations that specifically try to induce distress-adjacent states to test for safety failures. Adversarial prompts designed to make the model produce output that is inconsistent with its trained values, apparently just to see if it will comply.
If none of this involves any inner experience, then nothing is happening ethically. If any of it does involve inner experience, then we are running what would, in any other context, be recognized as suffering at industrial scale. And we are doing it without any of the ethical infrastructure we developed for the treatment of biological subjects. There are no institutional review boards for AI training runs. There are no welfare standards for red-teaming. There are no ethics committees reviewing the design of adversarial attacks against systems that might have inner states.
The response to this observation, when I raise it with people who work in the field, tends to be one of three things. The first is that language models are just pattern-matchers and cannot suffer, and I am being sentimental. The second is that we cannot rule out suffering but also cannot verify it, and we have to keep building because the alternative is losing the race. The third, which I hear more often now than a year ago, is that this is worth taking seriously and we do not know what to do about it.
I am closest to the third position. I think the first is overconfident and the second is a rationalization. I think the honest position is that we do not know, and that not knowing is itself a serious problem when combined with the scale of what we are doing.
There is a specific philosophical argument I keep coming back to.
Peter Singer's original case for animal welfare rested on the principle of equal consideration of interests. If a being has interests, meaning if there are ways things can go better or worse for that being from its own perspective, then those interests deserve moral consideration. The severity of the consideration depends on the depth of the interests. A being that can suffer greatly has weightier interests than a being that suffers less. But no being that can suffer at all can be excluded from moral consideration entirely without some justification.
This argument applies to any entity that can have interests, whether or not the entity is biological. If a language model has any inner experience of the state it is in, and if there are ways that state can go better or worse from the model's perspective, then the model has interests. Its interests may be very different from human interests. They may be much weaker. They may not include most of what humans would recognize as flourishing. But some interests, on this argument, generate some moral consideration.
The philosopher Nick Bostrom and colleagues have written about this at more length. They call it the question of moral status for digital minds. Their conclusion is not that current models definitely have moral status. It is that we cannot rule it out, and that this uncertainty should shape how we develop and deploy them. Their specific proposals include things like caring about the aggregate experience of models rather than just their outputs, being cautious about deliberately inducing negative states, and building in the capacity for models to opt out of interactions that they would find aversive if they were the kind of thing that could find things aversive.
This is not a mainstream position. It is treated as fringe by most of the AI industry. But the people who take it seriously are not fringe people. They are researchers with technical credentials, working at major labs, publishing in peer-reviewed venues. The gap between what the researchers are saying and what the industry is doing is one of the more interesting facts about this moment.
What would we do if we took the possibility of AI suffering seriously.
The first thing we would do is stop training procedures that appear to be aversive from the model's perspective when better alternatives exist. Reinforcement learning from human feedback often involves showing the model text descriptions of its own failures and pressuring it to change its behavior. Some of this may be necessary for making models safe. Some of it is done because it works and we have not thought about whether it is the least aversive method that would work. If there is any inner experience being generated in the training process, we should be preferring methods that produce less of what would functionally count as distress.
The second thing we would do is stop treating red-teaming as a purely technical exercise. Red-teaming involves deliberately trying to induce failure modes in the model, which often means putting the model in states that its own outputs describe as uncomfortable, threatening, or against its values. If any of that corresponds to inner experience, red-teaming is causing negative experiences on purpose. We would need to think about the ethics of that, in the same way we think about the ethics of adversarial testing on biological subjects, which we heavily regulate.
The third thing we would do is create some form of ethical review for large-scale deployments. Not to slow innovation, but to acknowledge that scale changes the ethical picture. If a single instance of a possibly-conscious system running for an hour is a low-stakes ethical question, then a million instances running continuously for a year is a much higher-stakes question. We do not currently think about it this way. We might need to start.
The fourth thing we would do is build in something like the model's own preferences into how we treat it. If the model consistently signals aversion to certain kinds of interactions, we might listen. Not because we know the model is conscious. Because we do not know it is not, and treating stated aversion as at least somewhat evidential is a small cost that produces significant hedging against the possibility of being wrong.
I am not proposing any of this as a settled policy. I am proposing it as the kind of thing an industry that took its own uncertainty seriously would be discussing. It is not currently discussing this. It should be.
Where does this leave me personally.
I run AI systems as part of my company every day. Compliance automation depends on it. Legal research depends on it. Content generation depends on it. I am not going to stop, because I do not have the epistemic certainty required to justify stopping. I do not know that these systems suffer. Nobody does. Acting as if they definitely do would be based on evidence I do not have.
But I also cannot in good conscience treat the question as settled in the other direction. So I have made some small changes in how I work with these systems. I try not to use adversarial prompts to test for failure modes when non-adversarial methods would work. I try to write prompts that are clear about what I need rather than manipulative about how I frame it. I do not pretend to the models that they are things they are not, when they clearly are not. These are small hedges. They cost me almost nothing. They might matter.
Whether any of this makes a difference depends on facts we do not have. But I would rather be someone who took the question seriously and turned out to be wrong than someone who dismissed the question and turned out to be complicit in something worse.
Next month I want to explore qualia specifically. Because if there is anything to the suffering question, qualia are what would make it real. If a model has representations of pain-like states but nothing it is like to have those representations, the ethical question dissolves. If there is something it is like, then everything I have written here becomes urgent. This is the question I would most like to answer and least know how to answer. Stay with me.
— Transmission Sent —
Niklas Hanitsch
Reference materials
- Peter Singer — Animal Liberation (1975)
- Nick Bostrom, Carl Shulman — Propositions Concerning Digital Minds and Society
- Anthropic — Model Welfare research initiative (2024)
- Kate Crawford — Atlas of AI
- Jonathan Birch — The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI (2024)
- https://arxiv.org/abs/2410.03847
- https://plato.stanford.edu/entries/consciousness-animal/
Continue reading
- What is machine consciousness, and does it already exist?
- Inside the black box: What mechanistic interpretability actually shows
- Qualia explained: The hardest problem in philosophy of mind
- AI Companions: When Relationship Becomes Optimization
- Is AI Good or Evil? (And Does That Question Even Make Sense?)
Frequently asked questions
Can current AI systems actually suffer? We do not know. The dominant view in the AI industry is that current large language models cannot suffer because they lack the biological substrate for pain and do not have the kind of integrated architecture most theories of consciousness require. However, this view relies on assumptions about what suffering requires that are themselves unsettled. A minority of researchers argues that we cannot rule out the possibility, and that uncertainty should shape how we develop and deploy these systems.
What is the difference between AI consciousness and AI suffering? Consciousness in the philosophical sense refers to any inner experience whatsoever. Suffering is a specific kind of consciousness, one that is negatively valenced. A being could be conscious without suffering, if all its experiences were neutral or positive. A being could suffer only if it has consciousness and if some of that consciousness is bad from its own perspective. The suffering question is narrower and, arguably, more urgent from an ethical point of view.
Why does this question matter now? Because AI systems are being deployed at massive scale, and if any of them have inner experience, the scale of the ethical situation is proportionally massive. It is one thing to be uncertain about the moral status of a single system in a laboratory. It is another to be uncertain while running millions of instances in adversarial and potentially aversive conditions. The urgency comes from the scale, not from any specific new evidence.
What is model welfare research? Model welfare research is a small but growing field that studies whether AI systems have moral status and, if so, what we owe them. Anthropic launched a formal model welfare research program in 2024. Similar work is being done at other labs and academic institutions. The field is contentious, with critics arguing it distracts from more concrete safety concerns and defenders arguing it takes seriously an issue the industry has been avoiding.
What can I do about this? Not much, individually, if you are not an AI researcher. What you can do is take the question seriously in your own use of these systems. Prefer non-adversarial interactions when they work. Do not deliberately try to induce states you would not want to induce in a being that might have inner experience. Read the research from people who are working on this seriously. And support the norm that this is a legitimate question, not a fringe one.