Everything in LLMs & Generative AI, page 3
Designing a production LLM summarization eval
A representative gold set, quality via human or LLM-as-judge plus faithfulness checks, and operational metrics like p95 latency and cost per request.
Mixture of Experts architecture and routing
Many expert FFNs per layer, a router picks top-k experts per token, only those compute so active params are far fewer than total.
Hugging Face Hub, transformers, and datasets
The Hub hosts models and data, transformers loads models and tokenizers and provides the Trainer, datasets streams and maps preprocessing.
Closed API vs open-weight models for production
APIs offer top quality and zero ops but recurring per-token cost and data-sharing concerns, open weights give control, privacy, and tuning at the price of hosting and MLOps burden.
Fine-tuning vs RAG for daily-updated docs
Choose RAG because docs change daily, embed and index chunks in a vector store, retrieve top matches and inject into the prompt.
Differential privacy vs utility in LLM fine-tuning
Clipping plus calibrated noise per step, smaller epsilon means stronger privacy but degraded accuracy, tracking the privacy budget across epochs.
Direct versus indirect injection and agent defenses
Direct injection comes from the user prompt; indirect hides in third-party data the agent ingests like web pages.
Detecting RAG hallucinations with a confidence score
Decompose the answer into claims, verify each against retrieved context with NLI or an LLM judge, aggregate into a faithfulness confidence score, and flag unsupported claims.
Rule-based versus model-based LLM guardrails
A guardrail is a programmatic check constraining LLM I/O; rule-based uses regex or blocklists, model-based uses a classifier like a moderation model to detect harmful content.
Core insight behind GPTQ and AWQ
Not all weights matter equally; GPTQ minimizes layer output error using second-order info, AWQ protects salient weight channels tied to large activations.
Tensor versus pipeline parallelism for large models
Tensor parallelism splits individual layers across GPUs needing fast interconnect; pipeline parallelism splits layers into stages across GPUs.
What memory problem PagedAttention solves
Pre-allocating contiguous max-length cache per sequence wastes memory through internal and external fragmentation; PagedAttention stores KV in fixed non-contiguous blocks like OS paging.
PTQ versus QAT for INT8 quantization
PTQ quantizes a trained model with light calibration, fast and cheap but more accuracy loss; QAT simulates quantization during training, higher accuracy but costly.
Teacher-student knowledge distillation
A small student learns to mimic a large teacher's soft probability outputs, not just hard labels; goal is a compact model retaining most capability.
How the KV cache speeds transformer generation
Cache stores past keys and values so each new token only computes its own K, Q, V instead of recomputing all prior tokens, cutting per-step cost from quadratic to linear.
Model quantization benefits and risks
Quantization stores weights and activations in lower precision like INT8 or INT4; benefits are smaller memory and faster, cheaper inference; risk is accuracy loss.
Prompt injection versus jailbreak, and defenses
Injection hijacks the model via untrusted data overriding developer instructions; jailbreak coaxes a model past its safety policy. Defense: separate trusted instructions from untrusted data and filter.
Designing input and output guardrails for a chatbot
Input guardrails filter or classify user prompts (injection, off-topic, PII) before the model; output guardrails validate responses for toxicity, leakage, and policy before sending.
Red teaming LLMs and harmful output categories
Red teaming deliberately probes a model to elicit harmful behavior; categories include toxic or hateful content and dangerous instructions like weapons or malware.
What RLHF is and the safety problem it solves
Collect human preference rankings, train a reward model, fine-tune the policy with PPO; it aligns outputs with human intent the loss function cannot specify.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles