More in AI & ML — page 4
Three techniques to cut LLM inference latency
WHAT IT TESTS: knowledge of software-level inference optimization. OUTLINE: quantization shrinks weights with small accuracy risk, KV-cache plus continuous batching boost throughput, speculative decoding drafts tokens for lossless speedup.
Multimodal video understanding architecture
WHAT IT TESTS: how vision and text fuse in multimodal models. OUTLINE: sample frames, encode them into visual tokens via a vision encoder and projector, concatenate with text tokens, let cross-attention fuse them.
Designing a production LLM summarization eval
WHAT IT TESTS: building task-specific eval beyond leaderboards. OUTLINE: a representative gold set, quality via human or LLM-as-judge plus faithfulness checks, and operational metrics like p95 latency and cost per request.
Mixture of Experts architecture and routing
WHAT IT TESTS: grasp of sparse activation and the gating router. OUTLINE: many expert FFNs per layer, a router picks top-k experts per token, only those compute so active params are far fewer than total.
Hugging Face Hub, transformers, and datasets
WHAT IT TESTS: practical fluency with the standard NLP toolchain. OUTLINE: the Hub hosts models and data, transformers loads models and tokenizers and provides the Trainer, datasets streams and maps preprocessing.
Closed API vs open-weight models for production
WHAT IT TESTS: balancing capability, cost, control, and compliance. OUTLINE: APIs offer top quality and zero ops but recurring per-token cost and data-sharing concerns, open weights give control, privacy, and tuning at the price of hosting and MLOps burden.
Fine-tuning vs RAG for daily-updated docs
WHAT IT TESTS: matching the right technique to freshness needs. OUTLINE: choose RAG because docs change daily, embed and index chunks in a vector store, retrieve top matches and inject into the prompt.
Differential privacy vs utility in LLM fine-tuning
WHAT IT TESTS: understanding DP-SGD's noise-for-privacy bargain. OUTLINE: clipping plus calibrated noise per step, smaller epsilon means stronger privacy but degraded accuracy, tracking the privacy budget across epochs.
Direct versus indirect injection and agent defenses
WHAT IT TESTS: agent security under injection. OUTLINE: direct injection comes from the user prompt; indirect hides in third-party data the agent ingests like web pages.
Detecting RAG hallucinations with a confidence score
WHAT IT TESTS: groundedness verification design. OUTLINE: decompose the answer into claims, verify each against retrieved context with NLI or an LLM judge, aggregate into a faithfulness confidence score, and flag unsupported claims.
Rule-based versus model-based LLM guardrails
WHAT IT TESTS: practical safety controls. OUTLINE: a guardrail is a programmatic check constraining LLM I/O; rule-based uses regex or blocklists, model-based uses a classifier like a moderation model to detect harmful content.
Core insight behind GPTQ and AWQ
WHAT IT TESTS: why advanced INT4 methods work. OUTLINE: not all weights matter equally; GPTQ minimizes layer output error using second-order info, AWQ protects salient weight channels tied to large activations.
Tensor versus pipeline parallelism for large models
WHAT IT TESTS: multi-GPU model sharding strategy. OUTLINE: tensor parallelism splits individual layers across GPUs needing fast interconnect; pipeline parallelism splits layers into stages across GPUs.
What memory problem PagedAttention solves
WHAT IT TESTS: KV-cache memory management at serving scale. OUTLINE: pre-allocating contiguous max-length cache per sequence wastes memory through internal and external fragmentation; PagedAttention stores KV in fixed non-contiguous blocks like OS paging.
PTQ versus QAT for INT8 quantization
WHAT IT TESTS: practical quantization trade-offs. OUTLINE: PTQ quantizes a trained model with light calibration, fast and cheap but more accuracy loss; QAT simulates quantization during training, higher accuracy but costly.
Teacher-student knowledge distillation
WHAT IT TESTS: grasp of model compression via distillation. OUTLINE: a small student learns to mimic a large teacher's soft probability outputs, not just hard labels; goal is a compact model retaining most capability.
How the KV cache speeds transformer generation
WHAT IT TESTS: understanding attention during decoding. OUTLINE: cache stores past keys and values so each new token only computes its own K, Q, V instead of recomputing all prior tokens, cutting per-step cost from quadratic to linear.
Model quantization benefits and risks
WHAT IT TESTS: deployment-efficiency fundamentals. OUTLINE: quantization stores weights and activations in lower precision like INT8 or INT4; benefits are smaller memory and faster, cheaper inference; risk is accuracy loss.
Prompt injection versus jailbreak, and defenses
WHAT IT TESTS: precise security distinctions. OUTLINE: injection hijacks the model via untrusted data overriding developer instructions; jailbreak coaxes a model past its safety policy. Defense: separate trusted instructions from untrusted data and filter.
Designing input and output guardrails for a chatbot
WHAT IT TESTS: layered safety architecture. OUTLINE: input guardrails filter or classify user prompts (injection, off-topic, PII) before the model; output guardrails validate responses for toxicity, leakage, and policy before sending.