Designing a production LLM summarization eval

WHAT THIS TESTS This checks whether you can move past generic leaderboards to an evaluation framework anchored in the actual production task, covering quality, latency, and cost together rather than a single number.

A GOOD ANSWER COVERS Start with a representative dataset drawn from real production documents, with reference summaries or rubrics, covering the length and domain distribution you actually serve. For quality, combine methods: human evaluation on a sample using a clear rubric for relevance, coherence, and conciseness; an LLM-as-judge for scale, calibrated against the human labels; and automated faithfulness checks that detect hallucinated or unsupported claims, since overlap metrics like ROUGE measure n-gram match, not factual accuracy. For operations, measure latency as p50 and p95 not just the mean, time to first token if streaming, throughput under concurrency, and cost per request derived from input and output token counts at the provider's rate. Use a held-out test set, run both candidates on identical inputs, and assess differences with statistical significance rather than anecdotes. Track refusal rate and failure modes too.

COMMON WRONG ANSWERS Using MMLU or other unrelated academic benchmarks; trusting ROUGE or BLEU alone; reporting only mean latency; ignoring cost; comparing on different inputs; declaring a winner without significance testing or a representative dataset.

LIKELY FOLLOW-UPS How do you calibrate an LLM judge against humans? Why p95 over mean latency? How do you detect hallucinations programmatically? How do you keep the eval set from leaking into training?

ONE CONCRETE EXAMPLE You collect five hundred real support tickets with human-written summaries, run both models on all of them, score outputs with a calibrated LLM judge plus a faithfulness classifier, and record p95 latency and per-request token cost. Model A scores slightly higher on quality but costs twice as much and has higher p95 latency, so the decision becomes an explicit quality-versus-cost trade rather than a benchmark ranking.