More in LLMs & Generative AI — page 4
Setting up an LLM-as-a-judge evaluation
WHAT IT TESTS: rigorous LLM-as-judge design. OUTLINE: define rubric, prefer pairwise comparison, randomize order, calibrate against humans; control position, verbosity, and self-preference bias.
Reference-free evaluation for open-ended dialogue
WHAT IT TESTS: knowing why n-gram overlap fails for dialogue. OUTLINE: ROUGE punishes valid paraphrases; use reference-free LLM-as-judge or learned scorers rating coherence, relevance, and groundedness. RED FLAG: defending ROUGE for open-ended generation.
How MMLU works and the contamination problem
WHAT IT TESTS: understanding of LLM benchmarks and their flaws. OUTLINE: MMLU is multiple-choice across 57 subjects scored by accuracy; contamination means test items leaked into pretraining, inflating scores.
How to evaluate a RAG system end to end
WHAT IT TESTS: ability to decompose RAG evaluation. OUTLINE: measure retrieval (recall, precision, MRR, NDCG) and generation (faithfulness, answer relevance) separately, plus end-to-end correctness. RED FLAG: judging only final answer quality with one number.
How FID is calculated versus Inception Score
WHAT IT TESTS: deep grasp of generative metrics. OUTLINE: FID fits Gaussians to Inception features of real and fake images then measures Frechet distance; it uses real references and detects mode collapse. RED FLAG: claiming IS uses real data.
Standard metric for image generation quality
WHAT IT TESTS: knowledge of generative model evaluation. OUTLINE: name FID, explain it compares feature distributions of real and generated images via a pretrained network. RED FLAG: relying only on eyeballing samples or pixel-level MSE.
Why human evaluation is the gold standard
WHAT IT TESTS: limits of automated metrics. OUTLINE: humans judge fluency, helpfulness, and correctness that n-gram or distribution metrics miss; automated scores correlate weakly with quality, are gameable, and penalize valid diverse outputs.
Perplexity versus BLEU for LMs
WHAT IT TESTS: choosing the right LM metric. OUTLINE: perplexity measures intrinsic next-token prediction quality needing no references; BLEU measures n-gram overlap with reference outputs for tasks like translation.
Batching strategy for multimodal training
WHAT IT TESTS: data strategy for mixed-modality training. OUTLINE: control dataset mixing ratios, use balanced sampling and per-source weighting, keep enough text-only data to avoid forgetting, and handle variable shapes via grouping or padding.
LLaVA versus Flamingo vision-LLM design
WHAT IT TESTS: comparing VLM integration strategies. OUTLINE: LLaVA projects image features into the LLM input space and feeds them as tokens, keeping the LLM mostly intact; Flamingo inserts gated cross-attention layers inside a frozen LLM.
Aligning text and image representations
WHAT IT TESTS: cross-modal representation alignment. OUTLINE: contrastive learning like CLIP pulls matched image-text pairs together and pushes mismatches apart; alternatively projection layers map one modality into a frozen model's space.
How Stable Diffusion generates images
WHAT IT TESTS: the three-part text-to-image pipeline. OUTLINE: the text encoder turns the prompt into embeddings, the U-Net predicts noise to remove conditioned on those embeddings, and the scheduler controls how noise is stepped down over iterations in…
Designing a Visual Question Answering system
WHAT IT TESTS: multimodal architecture design. OUTLINE: encode the image with a vision backbone, encode the question with a text encoder, fuse them via cross-attention into a joint representation, then decode or classify the answer.
Early versus late modality fusion
WHAT IT TESTS: how multimodal models combine inputs. OUTLINE: early fusion merges raw or low-level features so the model learns cross-modal interactions, while late fusion processes each modality separately and combines outputs.
Classic image captioning architecture
WHAT IT TESTS: the encoder-decoder captioning pattern. OUTLINE: a CNN encoder extracts image features, a recurrent or transformer decoder generates the caption word by word, and attention lets the decoder focus on image regions per word.
Temporal consistency in video diffusion
WHAT IT TESTS: extending image diffusion to video. OUTLINE: add temporal layers, such as temporal attention or 3D convolutions across frames, so the model attends across time and frames denoise jointly rather than independently.
Diffusion-based image inpainting design
WHAT IT TESTS: adapting diffusion models to inpainting. OUTLINE: at each denoising step keep the known region by replacing it with the noised original, let the model generate only the masked area, condition on prompt and mask.
DDIM: faster diffusion sampling
WHAT IT TESTS: understanding of accelerated diffusion sampling. OUTLINE: DDIM defines a non-Markovian deterministic process sharing DDPM's training, letting you skip steps and sample in far fewer iterations.
Evaluating image generation: FID and IS
WHAT IT TESTS: knowledge of generative image metrics. OUTLINE: FID compares feature distributions of real and generated images, lower is better; Inception Score rewards confident, diverse classes but ignores real data.
Designing an agent that resolves ambiguity
WHAT IT TESTS: agent design for under-specified requests. OUTLINE: detect ambiguity, gather evidence with the contact API, resolve relative time deterministically, ask the user only when genuinely uncertain, then confirm before the irreversible booking.