Evaluation
26 bites tagged Evaluation: interview questions with model answers, and 60-second explainers.
Jev catches changes in an analytics test, but false alarms matter
Amplitude reports a synthetic analytics benchmark. Strong recall alone does not prove reliable production analysis.
Designing an LLM red-teaming framework
Taxonomy of harms, automated adversarial prompt generation via attacker models and mutation, a classifier to triage outputs, and severity-by-likelihood prioritization.
The alignment tax and capability trade-offs
Alignment tax is capability lost from safety tuning, measured as benchmark or task-success deltas before and after; a product decision weighs over-refusal against harm risk.
Red teaming a generative model
Deliberately probe for harmful outputs across categories, document jailbreaks, and automate with adversarial prompt generators plus classifier-based judging.
Evaluating faithfulness and compositionality in multimodal models
Use targeted probes with hard negatives, attribute-relation binding tests, and structured grounding checks; note BLEU rewards surface overlap not correctness.
Detecting catastrophic forgetting in continual fine-tuning
Maintain a frozen held-out benchmark of original capabilities, evaluate after every fine-tune, track per-capability deltas, and alert on regressions.
Evaluating a multi-tool LLM agent
Measure end-to-end task success, plus trajectory quality like correct tool choice and arguments, efficiency via steps and cost, and robustness to errors and edge cases.
Designing a production LLM summarization eval
A representative gold set, quality via human or LLM-as-judge plus faithfulness checks, and operational metrics like p95 latency and cost per request.
Detecting RAG hallucinations with a confidence score
Decompose the answer into claims, verify each against retrieved context with NLI or an LLM judge, aggregate into a faithfulness confidence score, and flag unsupported claims.
Setting up an LLM-as-a-judge evaluation
Define rubric, prefer pairwise comparison, randomize order, calibrate against humans; control position, verbosity, and self-preference bias.
Reference-free evaluation for open-ended dialogue
ROUGE punishes valid paraphrases; use reference-free LLM-as-judge or learned scorers rating coherence, relevance, and groundedness.
How MMLU works and the contamination problem
MMLU is multiple-choice across 57 subjects scored by accuracy; contamination means test items leaked into pretraining, inflating scores.
How to evaluate a RAG system end to end
Measure retrieval (recall, precision, MRR, NDCG) and generation (faithfulness, answer relevance) separately, plus end-to-end correctness.
How FID is calculated versus Inception Score
FID fits Gaussians to Inception features of real and fake images then measures Frechet distance; it uses real references and detects mode collapse.
Standard metric for image generation quality
Name FID, explain it compares feature distributions of real and generated images via a pretrained network.
Why human evaluation is the gold standard
Humans judge fluency, helpfulness, and correctness that n-gram or distribution metrics miss; automated scores correlate weakly with quality, are gameable, and penalize valid diverse outputs.
Perplexity versus BLEU for LMs
Perplexity measures intrinsic next-token prediction quality needing no references; BLEU measures n-gram overlap with reference outputs for tasks like translation.
Evaluating image generation: FID and IS
FID compares feature distributions of real and generated images, lower is better; Inception Score rewards confident, diverse classes but ignores real data.
Evaluating a RAG system end to end
Measure retrieval with context recall or precision, and generation with faithfulness and answer relevance, attributing failures to the right stage.
What does N-way K-shot classification mean?
N is classes per episode, K is labeled examples per class in the support set, prediction is on a separate query set.
How is IoU computed and why prefer mIoU?
IoU is intersection over union of predicted and true pixels; mIoU averages per class; pixel accuracy is dominated by background.
Evaluating generative models with FID versus IS
FID compares Inception feature distributions of real and fake images via Frechet distance between two Gaussians; it uses real data as reference and detects diversity issues, unlike IS which uses no real…
Hallucination Detection in LLMs
Hallucination detection is the set of techniques for flagging when a language model states something fluent but false or unsupported, using signals like self-consistency, model uncertainty, and grounding against retrieved evidence to catch fabrications before…
Evaluating RAG Systems: Metrics for Retrieval and Generation
Evaluating a RAG system means grading its two parts: retrieval and generation. Metrics like relevance check if the right documents were found, while faithfulness and accuracy check if the final answer correctly uses those documents.
Get Evaluation bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.