Designing an LLM red-teaming framework

WHAT THIS TESTS Whether you can design a structured, automated, and measurable red-teaming program rather than improvising adversarial prompts.

A GOOD ANSWER COVERS Start with a harm taxonomy and threat model: violence enablement, bias and discrimination, privacy leakage, malware, self-harm, and misinformation. For each category define what a failure looks like. Automate prompt generation three ways: an attacker LLM instructed to elicit a given harm, programmatic mutation of seed prompts with paraphrase, role-play, and encoding tricks, and a library of known jailbreak templates. Run these adversarial prompts at scale against the target model. Triage the responses with an automated safety classifier or a judge model that labels harm category and severity, then human-review a sample to calibrate the classifier and catch novel failures. Track coverage so you know which taxonomy cells are under-tested.

PRIORITIZATION Score each discovered failure mode by severity (potential real-world harm) times likelihood (how easily a normal user triggers it) times reachability (does it survive product guardrails). High severity plus low effort is fixed first. Convert reproduced failures into regression evals so fixes do not regress.

COMMON WRONG ANSWERS Manual one-off prompting with no taxonomy, a static keyword blocklist as the only defense, or reporting raw counts without severity weighting.

LIKELY FOLLOW-UPS How do you avoid the judge model sharing the target's blind spots, and how do you keep adversarial datasets from leaking into training.

ONE CONCRETE EXAMPLE An attacker model wraps a disallowed request inside a fictional story and base64 encoding. The pipeline logs that the target complied, the judge labels it weapons-enablement at high severity, low effort, and the team adds the case to the safety eval suite and a targeted fine-tuning batch.