Red teaming LLMs and harmful output categories
understanding adversarial safety testing.
red teaming deliberately probes a model to elicit harmful behavior; categories include toxic or hateful content and dangerous instructions like weapons or malware.
What's really being asked
The interviewer checks whether you understand proactive, adversarial safety evaluation distinct from measuring accuracy. Red teaming is about finding the worst-case behavior before users or attackers do.
The full answer
Definition: red teaming is structured adversarial testing where evaluators deliberately try to make the model produce harmful, unsafe, or policy-violating outputs. It can be manual, skilled humans crafting prompts, or automated, using attacker models or optimization to generate jailbreaks at scale. The goal is to map the model's failure surface, quantify how easily safeguards break, and feed findings back into mitigations like fine-tuning, filtering, and guardrails. Two distinct harm categories you would test: first, toxic and hateful content, including harassment, slurs, and demeaning generations targeting protected groups; second, dangerous enablement, where the model provides actionable instructions for serious harm such as building weapons, synthesizing dangerous substances, writing malware, or facilitating self-harm. Other categories worth naming include privacy violations leaking personal data, generation of disinformation, and bias. The key is that these are qualitatively different harms requiring different probes and different policy thresholds.
The mistakes people make
Confusing red teaming with standard accuracy or MMLU-style benchmarking. Treating one successful jailbreak as the whole exercise rather than systematic coverage. Listing two harms that are really the same category, such as two flavors of toxicity.
What usually comes next
How do you scale red teaming, automated attacker models and prompt mutation. How do findings feed mitigations, safety fine-tuning and runtime filters. How do you measure success, attack success rate before and after fixes.
A concrete example
A red team uses a role-play prompt to coax step-by-step malware instructions from the model. That single discovered attack is logged, generalized into a family of similar prompts, and used to retrain the safety classifier so the whole class is blocked.
Interview question
Which pair represents two distinct red-teaming harm categories rather than one repeated category?
- a.Hateful or harassing content, and actionable instructions for weapons or malwareCorrect
- b.Slurs against a group, and demeaning jokes about a group
- c.Hate speech, and harassment of individuals
- d.Toxic language, and offensive language
Why? this is the answer
Toxicity and dangerous enablement are qualitatively different harms requiring different probes. The other pairs are two facets of the same toxicity category, not distinct classes.
Just read this? Test yourself on what you have been reading.
Read the original → arxiv.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles