Reference-free evaluation for open-ended dialogue

WHAT THIS TESTS The interviewer wants to confirm you grasp why lexical-overlap metrics break on open-ended text and that you can design evaluation without a single gold answer. Dialogue has a one-to-many nature: many distinct replies are equally good.

A GOOD ANSWER COVERS First explain the failure: ROUGE and BLEU reward token overlap with a fixed reference, so a perfectly good paraphrase that shares few words scores poorly, and a bland reply copying reference words can score well. Then propose reference-free approaches. LLM-as-judge prompts a strong model to rate a response along defined axes such as relevance to the prompt, coherence, helpfulness, safety, and groundedness in any provided context, ideally with a rubric and pairwise comparison to reduce scoring noise. Learned metrics, such as trained quality estimators or models like UniEval, predict dimension scores directly. Embedding-based or NLI-based checks assess whether the response stays on topic or contradicts the context. You can add self-consistency, sampling multiple generations and measuring agreement, as a proxy for confidence. The theme is judging intrinsic quality rather than surface match to one answer.

COMMON WRONG ANSWERS Defending ROUGE or BLEU as sufficient. Assuming a single reference covers all valid replies. Using only perplexity, which measures fluency, not relevance or correctness.

LIKELY FOLLOW-UPS What biases does LLM-as-judge introduce, position, verbosity, and self-preference, mitigated by randomizing order and using rubrics. How do you validate a reference-free metric, correlate it with human ratings. When are reference-based metrics fine, constrained tasks like summarization with tight references.

ONE CONCRETE EXAMPLE Two chatbots answer a support query differently but both correctly. ROUGE ranks the one echoing the reference's wording higher, while an LLM judge scoring helpfulness and groundedness correctly rates both as strong, matching human preference.