Teacher-student knowledge distillation

WHAT THIS TESTS The interviewer wants to confirm you understand distillation as a transfer of learned behavior, not just retraining a smaller model on the same labels. The soft-target insight is the crux.

A GOOD ANSWER COVERS The paradigm: a large, high-capacity teacher model is already trained and accurate but expensive to serve. A smaller student model is trained to imitate the teacher. The primary goal is compression and efficiency, producing a model that is faster, cheaper, and lighter to deploy while preserving as much of the teacher's accuracy as possible. The key data: rather than training the student only on hard one-hot ground-truth labels, you train it on the teacher's soft outputs, the full probability distribution over classes or tokens, often produced with a temperature that softens the distribution. These soft targets encode the teacher's relative confidence and the similarity structure between classes, the dark knowledge, which is far more informative than a single correct label. The student loss typically blends a distillation term, matching the teacher's soft outputs, with an optional supervised term on true labels. For LLMs the student often trains on teacher-generated text or token-level distributions over a large unlabeled or prompt-driven corpus.

COMMON WRONG ANSWERS Saying the student only uses hard labels, missing soft targets and dark knowledge. Confusing distillation with quantization or pruning. Claiming the goal is higher accuracy than the teacher, the realistic goal is retaining most accuracy at lower cost.

LIKELY FOLLOW-UPS Why use temperature, it reveals inter-class similarity. What is dark knowledge, the information in non-target probabilities. Can the student exceed the teacher, rarely, usually it approaches it. How does this differ from quantization, distillation changes architecture, quantization changes precision.

ONE CONCRETE EXAMPLE DistilBERT is trained to match BERT's softened output distribution, yielding a model roughly 40 percent smaller and significantly faster while retaining most of BERT's language-understanding performance, illustrating the compression goal.