Setting up an LLM-as-a-judge evaluation

WHAT THIS TESTS This is a senior-level design question: can you use a strong model as an evaluator without inheriting its blind spots. The interviewer is listening for both a concrete protocol and an honest account of bias.

A GOOD ANSWER COVERS Setup: define the task and a precise rubric with named criteria such as correctness, relevance, completeness, and safety, plus a scale or, better, a pairwise win-or-tie format that is more reliable than absolute numeric scores. Provide the judge the prompt, the candidate answers, any reference context, and explicit grading instructions; optionally request a brief chain-of-thought before the verdict to improve consistency. Run each comparison twice with the two answers in swapped positions to neutralize position bias and average. Validate the judge by correlating its verdicts with a human-labeled set, reporting agreement, and only trust the judge where agreement is high. Biases to control: position bias, the judge favors the first or second answer; verbosity bias, longer or more confident answers win regardless of quality; self-preference bias, a judge favors outputs matching its own style or its own model family; and leniency or score compression. Mitigations include order randomization, length normalization or instructing the judge to ignore length, using a different model family as judge, and rubric anchoring with examples.

COMMON WRONG ANSWERS Taking judge scores at face value without human calibration. Always presenting your model first. Letting the same model judge its own outputs without acknowledging self-preference. Using a vague good-or-bad prompt.

LIKELY FOLLOW-UPS How do you detect position bias, results flip when order swaps. Why pairwise over pointwise, humans and models compare more reliably than they score absolutely. How to handle ties and low inter-judge agreement.

ONE CONCRETE EXAMPLE In an A-versus-B test, the judge picks answer A 80 percent of the time. After swapping order, it picks the first slot 80 percent regardless of content, exposing position bias; averaging swapped runs corrects the ranking.