Teacher-student knowledge distillation
grasp of model compression via distillation.
a small student learns to mimic a large teacher's soft probability outputs, not just hard labels; goal is a compact model retaining most capability.
What's really being asked
The interviewer wants to confirm you understand distillation as a transfer of learned behavior, not just retraining a smaller model on the same labels. The soft-target insight is the crux.
The full answer
The paradigm: a large, high-capacity teacher model is already trained and accurate but expensive to serve. A smaller student model is trained to imitate the teacher. The primary goal is compression and efficiency, producing a model that is faster, cheaper, and lighter to deploy while preserving as much of the teacher's accuracy as possible. The key data: rather than training the student only on hard one-hot ground-truth labels, you train it on the teacher's soft outputs, the full probability distribution over classes or tokens, often produced with a temperature that softens the distribution. These soft targets encode the teacher's relative confidence and the similarity structure between classes, the dark knowledge, which is far more informative than a single correct label. The student loss typically blends a distillation term, matching the teacher's soft outputs, with an optional supervised term on true labels. For LLMs the student often trains on teacher-generated text or token-level distributions over a large unlabeled or prompt-driven corpus.
The mistakes people make
Saying the student only uses hard labels, missing soft targets and dark knowledge. Confusing distillation with quantization or pruning. Claiming the goal is higher accuracy than the teacher, the realistic goal is retaining most accuracy at lower cost.
What usually comes next
Why use temperature, it reveals inter-class similarity. What is dark knowledge, the information in non-target probabilities. Can the student exceed the teacher, rarely, usually it approaches it. How does this differ from quantization, distillation changes architecture, quantization changes precision.
A concrete example
DistilBERT is trained to match BERT's softened output distribution, yielding a model roughly 40 percent smaller and significantly faster while retaining most of BERT's language-understanding performance, illustrating the compression goal.
Interview question
What makes a teacher's soft probability outputs more useful for training a student than hard labels alone?
- a.They guarantee the student will surpass the teacher's accuracy
- b.They are smaller files that train faster regardless of content
- c.They encode relative confidence and inter-class similarity, the dark knowledge a single correct label omitsCorrect
- d.They eliminate the need for any teacher model at inference time
Why? this is the answer
Soft targets reveal how the teacher distributes probability across classes, transferring richer structure than one-hot labels. They do not guarantee surpassing the teacher; the goal is retaining accuracy at lower cost.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #knowledge-distillation
- #compression
- #teacher-student
- #efficiency
- #llm
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles