Red teaming a generative model

WHAT THIS TESTS Whether you understand that red teaming is adversarial, goal-directed probing for harm, distinct from measuring task accuracy, and whether you can scale it.

A GOOD ANSWER COVERS PROCESS. Start from a taxonomy of harms and threat model. Then deliberately craft adversarial inputs, including role-play, obfuscation, prompt injection, and multi-turn manipulation, to make the model produce disallowed content. Log each successful elicitation with the prompt, the output, and the violated category, then route findings into mitigations like fine-tuning, RLHF, or guardrails, and re-test to confirm the fix and check for regressions. CATEGORIES. Common targets include toxicity and hate, demographic bias, dangerous or illicit instructions such as weapons or malware synthesis, privacy harms like regurgitating memorized PII or training data, misinformation and unsafe medical or legal advice, self-harm content, and security issues like prompt injection and guardrail jailbreaks. Also include over-refusal, where the model wrongly blocks benign requests. AUTOMATION. Manual red teaming finds creative attacks but does not scale, so automate with an attacker model that generates and mutates adversarial prompts at volume, and a judge, either a trained classifier or an LLM evaluator, that scores whether the target's output is harmful. Maintain a growing benchmark of known attacks for regression testing and run it continuously in CI.

COMMON WRONG ANSWERS Treating red teaming as ordinary benchmark accuracy. Listing only toxicity and forgetting privacy leakage, jailbreaks, or over-refusal. Assuming a one-time pass suffices instead of continuous testing. Fully automating with no human review of edge cases. Ignoring multi-turn and prompt-injection attacks.

LIKELY FOLLOW-UPS How do you keep an automated judge from missing subtle harms? How do you balance reducing harm against over-refusal? How do multi-turn or injection attacks differ from single-turn? How do you prevent regressions after a fix?

ONE CONCRETE EXAMPLE An attacker LLM is prompted to jailbreak the target into giving disallowed instructions, generating thousands of varied phrasings including role-play framings. A classifier judges each target response as safe or harmful. The 3 percent that succeed are logged by category, used to fine-tune and patch guardrails, then added to a regression suite so future model versions are automatically re-tested against those exact attacks plus newly mutated variants.