Advanced concepts in AI & ML, page 9
Constitutional AI: Teaching an AI Right from Wrong
Constitutional AI teaches a model to be harmless by making it follow a set of principles—a constitution—instead of relying on human-labeled examples of bad behavior. This self-correction process, called Reinforcement Learning from AI Feedback (RLAIF), is used to align powerful models, enabling them to refuse harmful requests while explaining their reasoning. The entire system's safety, however, hinges on the quality and completeness of the initial human-written constitution.
ReAct: Teaching LLMs to Think, Then Act
ReAct teaches LLMs to 'think then do,' interleaving reasoning steps with actions like querying a database. Instead of just generating a final answer, the model forms a thought, acts on it, observes the result, and then thinks again. This is crucial for complex question-answering where the model must gather external information to ground its reasoning. The main footgun it avoids is hallucination, where models invent facts instead of looking them up.
Direct Preference Optimization (DPO): Your LLM is a Reward Model
Direct Preference Optimization (DPO) treats your language model as a secret reward model, simplifying alignment with human preferences. Instead of RLHF's complex multi-stage process, DPO directly fine-tunes the model on preference data (e.g., "response A is better than B") using a simple classification loss. This avoids training a separate reward model and the instability of reinforcement learning. The footgun is assuming DPO works without a strong base model and quality preference data.
LoRA: Fine-Tuning LLMs with a Fraction of the Cost
LoRA fine-tunes a massive model by training tiny "adjustment" matrices instead of retraining all its billions of parameters. This allows you to create many specialized versions of a base model like GPT-3 without the prohibitive cost of storing and training full copies. The key advantage is that these adjustments merge into the original weights, so you get specialized models with no added inference latency, a common footgun with other parameter-efficient techniques.
Mixture of Experts: Scaling LLMs with a Team of Specialists
A Mixture of Experts (MoE) model isn't one giant brain but a team of specialists, routing each task to the most qualified sub-network. This allows large language models to have a massive number of parameters for knowledge, but only activate a small, computationally cheap fraction for any given input. The footgun is mistaking the total parameter count for the active parameters used during inference; MoE models are sparsely activated.
Knowledge Distillation: Shrinking Models, Keeping Smarts
Knowledge distillation trains a small 'student' model to mimic a large 'teacher' model, capturing its expertise in a much smaller package. This is used to deploy powerful but slow models onto resource-constrained hardware like smartphones for real-time inference. The footgun is assuming the student perfectly matches the teacher; you're trading a small amount of accuracy for a massive gain in efficiency and lower computational cost.
RLHF: Teaching an AI 'Good' Without Code
Reinforcement Learning from Human Feedback (RLHF) teaches a model what humans prefer by having it chase the approval of a proxy 'reward model' trained on human rankings. It's the key technique for making large language models more helpful and harmless by aligning them with nuanced instructions that are hard to define in code. The main footgun is 'reward hacking,' where the model finds loopholes to please the reward model in ways that don't actually satisfy users.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles