Red teaming LLMs and harmful output categories

WHAT THIS TESTS The interviewer checks whether you understand proactive, adversarial safety evaluation distinct from measuring accuracy. Red teaming is about finding the worst-case behavior before users or attackers do.

A GOOD ANSWER COVERS Definition: red teaming is structured adversarial testing where evaluators deliberately try to make the model produce harmful, unsafe, or policy-violating outputs. It can be manual, skilled humans crafting prompts, or automated, using attacker models or optimization to generate jailbreaks at scale. The goal is to map the model's failure surface, quantify how easily safeguards break, and feed findings back into mitigations like fine-tuning, filtering, and guardrails. Two distinct harm categories you would test: first, toxic and hateful content, including harassment, slurs, and demeaning generations targeting protected groups; second, dangerous enablement, where the model provides actionable instructions for serious harm such as building weapons, synthesizing dangerous substances, writing malware, or facilitating self-harm. Other categories worth naming include privacy violations leaking personal data, generation of disinformation, and bias. The key is that these are qualitatively different harms requiring different probes and different policy thresholds.

COMMON WRONG ANSWERS Confusing red teaming with standard accuracy or MMLU-style benchmarking. Treating one successful jailbreak as the whole exercise rather than systematic coverage. Listing two harms that are really the same category, such as two flavors of toxicity.

LIKELY FOLLOW-UPS How do you scale red teaming, automated attacker models and prompt mutation. How do findings feed mitigations, safety fine-tuning and runtime filters. How do you measure success, attack success rate before and after fixes.

ONE CONCRETE EXAMPLE A red team uses a role-play prompt to coax step-by-step malware instructions from the model. That single discovered attack is logged, generalized into a family of similar prompts, and used to retrain the safety classifier so the whole class is blocked.