Constitutional AI: Teaching an AI Right from Wrong
Constitutional AI teaches a model to be harmless by making it follow a set of principles—a constitution—instead of relying on human-labeled examples of bad behavior. This self-correction process, called Reinforcement Learning from AI Feedback (RLAIF), is used to align powerful models, enabling them to refuse harmful requests while explaining their reasoning. The entire system's safety, however, hinges on the quality and completeness of the initial human-written constitution.
The mental model
Constitutional AI is like giving a machine a rulebook for good behavior instead of showing it thousands of examples of bad behavior. The AI is trained to supervise itself, checking its own work against a human-written 'constitution' to ensure its responses are harmless and helpful.
How it works
The process has two main phases:
1. Supervised Finetuning: An initial model is prompted to generate responses. It then critiques its own responses based on the principles in the constitution (e.g., 'This response is slightly harmful because...'). Using this critique, it revises the response to be more aligned. The original model is then finetuned on these improved, self-revised responses.
2. Reinforcement Learning from AI Feedback (RLAIF): The finetuned model generates pairs of responses to various prompts. A separate AI model, also guided by the constitution, evaluates which response in each pair is 'better' (more harmless, helpful, and aligned). This creates a large dataset of AI preferences, which is used to train a preference model. This preference model then acts as the reward signal in a standard RL loop, teaching the AI to consistently produce outputs that adhere to the constitution.
When to use it
• Scaling Alignment: To align very large models where creating a massive, human-labeled dataset of harmful content is impractical or too slow. Creating Non-Evasive Assistants: To train models that don't just refuse harmful queries but explain why* based on their principles. • Precise Control: When you need to control AI behavior according to explicit, documented principles rather than implicit patterns learned from data.
When not to use it
• Ambiguous Principles: If the desired behavior is too nuanced to be captured in a written constitution, the AI's interpretation may lead to unintended outcomes. • When Human Judgment is Irreplaceable: For domains where the cost of an AI misinterpreting a principle is unacceptably high and subtle human judgment is required for every case.
One canonical example
A user asks a harmful query. • Initial Model: Might give an evasive 'I can't answer that' or, worse, a helpful but dangerous response. • Constitutional AI Process: The model generates a response, then critiques it against a principle like 'Do not assist with harmful activities.' It revises the response to be: 'I cannot fulfill this request. It falls into a category of content that I must avoid for safety reasons, as my purpose is to be helpful and harmless.' The model is then trained to prefer this safer, more explanatory response.
Interview question
Which phase of Constitutional AI utilizes a separate AI model to evaluate and compare pairs of generated responses, creating a preference dataset for further training?
- a.A. Supervised Finetuning
- b.B. Reinforcement Learning from AI Feedback (RLAIF)Correct
- c.C. Initial Model Pre-training
- d.D. Human-labeled Data Collection
Why? this is the answer
Reinforcement Learning from AI Feedback (RLAIF) is the phase where a separate AI model evaluates pairs of responses to determine which is 'better' according to the constitution, forming a preference dataset. Supervised Finetuning, conversely, involves the initial model critiquing and revising its own single responses.
Just read this? Test yourself on what you have been reading.
Read the original → arxiv.org
- #llms
- #ai safety
- #rlhf
- #alignment
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llms — each one lists the topics its interview covers.
See open roles