Constitutional AI: Teaching an AI Right from Wrong

### The mental model Constitutional AI is like giving a machine a rulebook for good behavior instead of showing it thousands of examples of bad behavior. The AI is trained to supervise itself, checking its own work against a human-written 'constitution' to ensure its responses are harmless and helpful.

### How it works The process has two main phases:

1. **Supervised Finetuning:** An initial model is prompted to generate responses. It then critiques its own responses based on the principles in the constitution (e.g., 'This response is slightly harmful because...'). Using this critique, it revises the response to be more aligned. The original model is then finetuned on these improved, self-revised responses.

2. **Reinforcement Learning from AI Feedback (RLAIF):** The finetuned model generates pairs of responses to various prompts. A separate AI model, also guided by the constitution, evaluates which response in each pair is 'better' (more harmless, helpful, and aligned). This creates a large dataset of AI preferences, which is used to train a preference model. This preference model then acts as the reward signal in a standard RL loop, teaching the AI to consistently produce outputs that adhere to the constitution.

### When to use it * **Scaling Alignment:** To align very large models where creating a massive, human-labeled dataset of harmful content is impractical or too slow. * **Creating Non-Evasive Assistants:** To train models that don't just refuse harmful queries but explain *why* based on their principles. * **Precise Control:** When you need to control AI behavior according to explicit, documented principles rather than implicit patterns learned from data.

### When NOT to use it * **Ambiguous Principles:** If the desired behavior is too nuanced to be captured in a written constitution, the AI's interpretation may lead to unintended outcomes. * **When Human Judgment is Irreplaceable:** For domains where the cost of an AI misinterpreting a principle is unacceptably high and subtle human judgment is required for every case.

### One canonical example A user asks a harmful query. - **Initial Model:** Might give an evasive 'I can't answer that' or, worse, a helpful but dangerous response. - **Constitutional AI Process:** The model generates a response, then critiques it against a principle like 'Do not assist with harmful activities.' It revises the response to be: 'I cannot fulfill this request. It falls into a category of content that I must avoid for safety reasons, as my purpose is to be helpful and harmless.' The model is then trained to prefer this safer, more explanatory response.