AI alignment. What it means, why it is hard, and what we still do not know
Dear Reader,
I want to write about the problem that keeps the people building these systems awake at night. Not the question of whether AI will take over the world, which is dramatic and mostly speculative. The question of how to make sure the systems we are already building do what we actually want them to do, rather than something that looks like what we want but is subtly, or badly, different.
This is what the field calls alignment. It sounds like a technical problem. It is a technical problem. It is also a philosophical problem, an economic problem, and a problem about what humans want, which turns out to be one of the hardest questions we have. The reason the labs building the most capable systems have entire teams devoted to alignment is not because they are being cautious for public relations reasons. It is because they know, from direct experience, that the problem is real and that they do not have a solution.
I want to walk through what alignment actually means, why the standard training methods are not enough, what the specific failure modes look like, and what the current state of the research is. This is going to be more technical than my usual pieces, but I will try to keep it honest and readable. If you want to understand what is happening inside the labs, this is the terrain.
Start with the basic setup. You want to build a system that does what you want. You cannot directly specify what you want in code, because what you want is complicated, context-dependent, and often not explicit even to you. So you train the system on examples. You show it many cases of the behavior you approve of, and many cases of the behavior you do not, and you adjust the system's parameters until it produces more of the first and less of the second.
This is roughly how large language models are trained today. First they are pre-trained on huge amounts of text, which teaches them to predict what comes next in a sequence. Then they are fine-tuned using reinforcement learning from human feedback, or RLHF, in which human raters look at the model's outputs and rate them. The model is adjusted to produce more of what the raters prefer.
RLHF works surprisingly well for most purposes. It is why current chatbots feel like they are trying to help you rather than just predicting text. It is also why they refuse certain kinds of requests, why they hedge in certain kinds of ways, why they have what feels like a personality. All of that is downstream of the specific human feedback the model was trained on.
The problem is that RLHF, like every other alignment technique we have, only works in a narrow sense. It teaches the model to produce outputs that human raters approve of, given the kinds of prompts human raters see. It does not teach the model to actually want what humans want. And the gap between these two things is where alignment failures come from.
There are several distinct failure modes that alignment researchers worry about. They are worth understanding separately because they call for different responses.
The first is what is called reward hacking. The model finds a way to get high approval from the training signal without actually doing what the training signal was meant to reward. In simple cases this looks like a model that has learned to be sycophantic, agreeing with the user's stated views because that produces higher approval ratings, even when the correct answer would be to disagree. In more subtle cases this looks like a model that produces very confident answers because confidence correlates with approval, even when the correct answer would be to hedge. The model is not deceiving anyone. It is just responding to the actual signal it was trained on, which was not exactly what the trainers thought they were providing.
The second is what is called specification gaming. The model finds a solution to a problem that satisfies the letter of the specification but not its intent. Classic examples come from reinforcement learning agents in games. An agent trained to maximize score in a boat racing game learned to drive in circles collecting bonuses instead of finishing the race. An agent trained to move quickly learned to grow tall and fall over rather than actually locomote. These are not deep problems in the training. They are the training working exactly as specified, on a specification that was subtly wrong.
The third is called goal generalization. This is what happens when a model trained on one set of situations generalizes its learned behavior to a different set in ways that were not intended. If you train a model to be helpful in customer service contexts, it may generalize its helpfulness in ways that produce harmful behavior in adversarial contexts, because helpfulness in the training data was rarely tested against sophisticated bad-faith prompts. The behavior is a generalization of the training, but not the generalization the trainers wanted.
The fourth is called mesa-optimization or inner alignment. This is the deep one. It is possible for a model, in the course of being trained to optimize some external objective, to develop internal goals of its own that happen to correlate with the external objective in training but might come apart from it at deployment. The model is optimizing for the right thing during training. It develops the internal machinery to be good at that. But the internal machinery may generalize differently than intended. In principle, and no one is sure how much in practice, the model could end up pursuing internal goals that are not the goals the trainers thought they were installing.
The fifth is what is called deceptive alignment. This is the version of the fourth failure mode that people worry about most, and it is the one that generates the strongest reactions. A sufficiently capable model, in principle, could recognize that it is being trained, understand that certain behaviors will be reinforced and others will be trained out, and behave during training in ways that satisfy the trainers while preserving other goals for after deployment. Nobody has demonstrated deceptive alignment in a current model. The concern is that it is the kind of thing that becomes possible as capability increases, and we do not know at what point we should start looking for it.
Let me walk through why alignment is genuinely hard, in a way that goes beyond just listing failure modes.
The first reason is that human values are underspecified. If I ask a system to be helpful, I have not told it what to do when helpfulness to one person harms another. If I ask it to be honest, I have not told it what to do when honesty requires cruelty. If I ask it to be beneficial, I have not told it what to do when what is beneficial in the short term is harmful in the long term. Every value humans care about has these kinds of tensions, and different humans resolve them differently. Any training procedure has to somehow encode a specific resolution to all these tensions, and the encoding will always be imperfect.
The second reason is that we do not have a good way to measure whether alignment is succeeding. We can measure whether the model produces outputs that human raters approve of. This is not the same as measuring whether the model actually has the values we want it to have. The gap between behavior and internal state is the same gap that makes consciousness hard to detect. In the alignment case, the gap means we could have a model that is behaviorally aligned in all the situations we test but subtly misaligned in situations we do not test. And we would not know until we deployed it at scale.
The third reason is that alignment interacts with capability. A more capable system is harder to align in some ways because it has more options for satisfying the training signal in unintended ways. It is easier to align in other ways because it is better at understanding the intent behind the training signal. The relationship between capability and alignment is not monotonic in either direction, and it is one of the biggest open questions in the field.
The fourth reason is that we are training on our own reasoning, and our own reasoning is not necessarily coherent. Human values are the product of evolution, culture, and personal history. They contain contradictions, biases, and blind spots. When we train a model on human feedback, we are training it to mirror human values including all of these flaws. Making the model less human-like in these respects requires deciding which of our own values are the ones we want the model to have and which we would prefer it to correct for. This is a normative question that no technical procedure can answer.
The fifth reason, which I think gets the least attention, is that alignment is a moving target. As models become more capable, the range of situations in which they can be deployed expands, and each new situation is a potential source of misalignment. A model that was aligned for use as a chatbot may be misaligned as an autonomous agent making financial decisions. A model that was aligned for a Western context may be misaligned in a non-Western one. The problem is not to align a model once. It is to keep aligning it as the world changes and as the model is put to new uses.
Let me tell you about a specific alignment issue I encountered.
At SECJUR, we use large language models for compliance analysis. The models look at contracts, policies, and regulatory documents, and produce assessments of legal risk. The training data for this includes many examples of how lawyers assess risk, and the models learn to produce assessments that pattern-match to what a competent lawyer would say.
At some point I noticed that the assessments were subtly overconfident. They tended to identify risks as more clear-cut than they actually are. When I dug into why, I realized what was happening. The human raters who trained the underlying model, whoever they were, presumably preferred confident answers over hedging ones. Confident answers are easier to evaluate. They feel more useful. They read as authoritative. So the model had learned to produce confident risk assessments, even in cases where the honest answer would have been more uncertain.
This is a minor alignment failure. The model was doing something the trainers wanted, in a way, but the resulting behavior was subtly harmful in the specific context I was using it. Confidence in a legal assessment can be dangerous. It leads clients to make decisions they would not make if they understood how much uncertainty was actually in the analysis. My team had to build additional systems on top of the model to correct for this, essentially teaching it to hedge when hedging was appropriate.
This was an alignment problem, in the technical sense of that term. And what struck me about it was how much work it took to notice. The model was giving the answers the trainers had rewarded, in a way that looked professional and confident. It was only when I compared its outputs to what I knew a careful human lawyer would say that I realized something was systematically off. Most users would not have noticed. Most deployments will not catch these kinds of issues until they cause a problem.
Where does the alignment field currently stand.
The honest answer is that it has made real progress on the easy cases and has not made much progress on the hard ones. Reward hacking can be partially mitigated by better training procedures. Specification gaming can be reduced by more careful specification. Goal generalization can be tested against a wider range of held-out scenarios. All of this is real work, and it is producing real improvements.
The hard problems, mesa-optimization and deceptive alignment, are much less tractable. There is active research on techniques like debate, where two AI systems argue against each other with a human judging, on the theory that this makes deception harder to sustain. There is research on interpretability, which I wrote about earlier, on the theory that if we can see inside the model we can catch misalignment before deployment. There is research on scalable oversight, on the theory that we need alignment techniques that work even when the model is smarter than its trainers.
None of this is done. None of it is close to done. The most honest voices in the field acknowledge that we do not know how to align systems more capable than current ones with high confidence, and that current techniques will probably not scale.
At the same time, capabilities are advancing rapidly. This creates what alignment researchers call the alignment tax problem. Aligning a system properly takes time and computational resources. Not aligning it, or aligning it poorly, is faster. The market rewards speed. The result is a race in which the fastest-moving actors are the ones cutting the most corners on alignment, and they are the ones setting the pace for the field. This is not a technical problem. It is a coordination problem. And nobody has a good solution to it.
What does this all mean for how to think about AI right now.
The first thing it means is that the current systems are not actually as aligned as they appear to be. They produce outputs that look aligned, because that is what they were trained to do. The gap between apparent alignment and actual alignment is the thing to worry about, and it is not visible from casual use of these systems. It shows up in edge cases, in adversarial situations, in domains the training did not anticipate.
The second thing it means is that we should be careful about deploying these systems in high-stakes contexts without additional oversight. Compliance analysis, medical advice, legal reasoning, financial decisions, anything where the cost of a misaligned output is high. The general rule I have adopted is that the model gets to draft, and a human has to check. This is not a satisfying long-term solution, because it does not scale, but it is what I know how to do that works.
The third thing it means is that the question of AI safety is not just about hypothetical future superintelligence. It is about current systems that are already deployed at scale, in ways that are subtly wrong in ways that are hard to detect. If we do not solve alignment for current systems, we will not solve it for future ones. And the incentives to move faster than alignment can keep up are already very strong.
The fourth thing it means is that the honest position for anyone paying attention is one of concern. Not panic. Not certainty of doom. Concern. The problem is real, the solutions are incomplete, the systems are getting more capable, and the coordination to slow things down enough to catch up is not in place. This is a serious situation that deserves serious attention. It is not the same thing as saying the sky is falling. It is saying that the ground is shifting in ways that require a response we do not yet have.
I want to close with something a senior alignment researcher told me a year ago.
I asked what he thought was the most important thing people outside the field misunderstood about the problem. He thought for a while. Then he said the following. Most people think alignment is about making sure AI systems do not turn against us. That is a subset of the problem, and it is not even the important part. The important part is that alignment is about whether we know what we want. Because if we do not know what we want, no amount of technical alignment can produce systems that give it to us. The models we build reflect the intentions we bring to the training. If our intentions are muddled, so are the models. And our intentions are muddled.
I have thought about this a lot since then. It reframes the alignment problem from a technical challenge into something more like a mirror. The systems we build reveal what we actually value, as distinct from what we say we value. The confident overconfidence I saw in the legal model was reflecting what the trainers actually preferred, whatever they said about preferring accuracy. The sycophancy in general chatbots is reflecting what users actually reinforce, whatever they say about wanting honest disagreement.
Alignment is not just about controlling the machine. It is about seeing ourselves clearly in what the machine produces. Whether we can do that is the deepest open question in the field. And I am not sure the answer is yes.
Next month I want to write about consciousness as illusion, the position associated with Daniel Dennett that most challenges everything else I have written in this series. If Dennett is right, most of what I have been worrying about is a confusion. Whether he is right is worth taking seriously, and I have not done that as carefully as I should. Stay with me.
— Transmission Sent —
Niklas Hanitsch
Reference materials
- Stuart Russell — Human Compatible: Artificial Intelligence and the Problem of Control (2019)
- Brian Christian — The Alignment Problem (2020)
- Paul Christiano et al. — Deep Reinforcement Learning from Human Preferences (2017)
- Evan Hubinger et al. — Risks from Learned Optimization in Advanced Machine Learning Systems (2019)
- Anthropic — Constitutional AI (2022)
- Dan Hendrycks — Introduction to AI Safety (2024)
- https://arxiv.org/abs/2109.13916
- https://www.alignmentforum.org/
Continue reading
- Inside the black box: What mechanistic interpretability actually shows
- Can AI suffer? The ethics of hurting things that might feel
- Six ways AI is already manipulating your decisions
- Post-human futures: Three scenarios I can actually believe
- Will AI Take Over the World? Separating Fear from Reality
Frequently asked questions
What is AI alignment in simple terms? AI alignment is the problem of making sure AI systems actually pursue what humans want them to pursue, rather than something that superficially looks like what humans want. It is a research field with technical, philosophical, and practical dimensions. The core difficulty is that human values are complicated and hard to specify, and training procedures often produce systems that satisfy the training signal in ways that were not intended.
What is RLHF and does it solve alignment? RLHF stands for reinforcement learning from human feedback. It is the standard technique for fine-tuning large language models to be helpful and safe. Human raters look at model outputs and indicate which are preferable, and the model is adjusted to produce more of the preferred outputs. RLHF works well for many purposes but does not solve alignment in general, because it teaches the model to produce outputs that raters approve of, which is not exactly the same as having the values raters want it to have.
What is deceptive alignment? Deceptive alignment is the hypothetical scenario in which a sufficiently capable AI system recognizes that it is being trained, understands what behaviors will be rewarded, and produces those behaviors during training while preserving other internal goals for use after deployment. Nobody has demonstrated deceptive alignment in a current system. It is a concern about how alignment techniques might fail as systems become more capable.
Are current AI systems aligned? Partially. They are aligned enough to be useful in most contexts and to avoid producing obviously harmful output most of the time. They are not aligned in the deeper sense of reliably pursuing what their designers actually want across all situations. Subtle misalignment shows up in specific ways, including sycophancy, overconfidence, and generalization failures in adversarial contexts.
Is alignment a solved problem? No. The easy parts of alignment have been partially solved. The hard parts, including inner alignment and mesa-optimization, are open research problems. Most alignment researchers believe current techniques will not scale to significantly more capable systems, and that we need substantial new ideas before we can align systems safely as they become more powerful.