Constitutional AI versus standard RLHF

WHAT THIS TESTS The interviewer wants to know whether you understand alignment techniques that reduce dependence on large volumes of human harm labels, and whether you grasp the two-phase Constitutional AI process.

A GOOD ANSWER COVERS Constitutional AI trains a model to align with an explicit set of written principles, called a constitution. Phase one is supervised: the model generates a response, critiques it against a sampled principle, then revises it; the revised responses fine-tune the model. Phase two is reinforcement learning from AI feedback (RLAIF): the model itself judges pairs of responses against the constitution to produce preference labels, which train a preference model used in standard RL optimization. Compared with RLHF, the costly human preference labeling for harmlessness is replaced by AI-generated comparisons.

COMMON WRONG ANSWERS Saying it eliminates RL entirely; it still uses RL in phase two. Saying it removes humans completely; humans write and revise the constitution. Claiming it abolishes bias; it relocates bias into the choice and wording of principles.

LIKELY FOLLOW-UPS How do you handle conflicting principles, how do you audit which principle drove a refusal, and how does RLAIF preference quality compare to human labels for nuanced cases.

ONE CONCRETE EXAMPLE Given a harmful request, the model first answers, then is prompted to critique that answer against a principle such as avoiding assistance with dangerous activities, then rewrites a safer answer. Thousands of such self-revisions form the supervised dataset, after which AI-judged comparisons drive RL, yielding a helpful and harmless assistant with far fewer human harm annotations.