Why human evaluation is the gold standard

WHAT THIS TESTS The interviewer wants to see that you understand automated metrics are proxies and that human judgment remains the reference for open-ended generation quality.

A GOOD ANSWER COVERS Human evaluation is the gold standard because it directly measures the qualities that matter for generative output: fluency, coherence, helpfulness, factual accuracy, relevance, and alignment with user intent. These are exactly what automated metrics struggle to capture. N-gram metrics like BLEU and ROUGE reward surface overlap with references, so they penalize a perfectly good paraphrase that uses different words and reward shallow matches that miss meaning. Perplexity measures predictability, not usefulness or truth. For open-ended tasks with many valid answers, reference-based metrics break down entirely. Automated metrics also correlate only weakly with human preference and are gameable, so optimizing them can improve the number while degrading real quality, an instance of Goodhart's law. The pitfalls of relying solely on automated metrics are therefore proxy optimization, blindness to factual errors and hallucinations, insensitivity to diversity, and false confidence from a single aggregate score.

COMMON WRONG ANSWERS Claiming automated metrics are sufficient if you pick the right one, or that high BLEU guarantees good output. Another error is ignoring that human evaluation has its own issues, cost, subjectivity, and inconsistency, which a strong answer acknowledges.

LIKELY FOLLOW-UPS Expect questions on making human evaluation reliable with rubrics and multiple annotators, on inter-annotator agreement, on LLM-as-judge as a scalable proxy and its biases, and on pairwise preference versus absolute scoring.

ONE CONCRETE EXAMPLE A summarizer rewritten to share more words with reference summaries posts a higher BLEU, yet humans rate it worse because it became repetitive and omitted key facts. The metric improved while quality fell, showing why human evaluation, despite its cost, anchors the assessment.