Model quantization benefits and risks

WHAT THIS TESTS The interviewer wants to confirm you understand a core technique for making large models deployable and that you can name both the upside and the honest downside. It is a practical efficiency question.

A GOOD ANSWER COVERS Definition: quantization reduces the numerical precision used to store and compute a model's parameters, mapping high-precision floating-point values, typically FP32 or FP16, to lower-precision representations such as 8-bit or 4-bit integers, using a scale and zero-point to map ranges. Benefits: a model in INT8 uses roughly a quarter of the memory of FP32 and INT4 about an eighth, which shrinks the footprint, reduces memory bandwidth, the usual bottleneck in LLM inference, lowers cost, and speeds up generation, often enabling a large model to fit on a single or smaller GPU. Risks: lower precision introduces rounding error that can degrade output quality, and LLMs are sensitive to outlier activations that dominate the range, so naive quantization at INT4 can cause sharp accuracy drops. The risk grows as bit-width falls and varies by layer; some layers are more sensitive than others. Good answers mention that quantization targets inference, not training, and that calibration or advanced schemes mitigate the loss.

COMMON WRONG ANSWERS Claiming quantization speeds up training, it primarily targets inference. Saying it is always lossless. Ignoring activation outliers as the main accuracy hazard. Confusing it with pruning or distillation.

LIKELY FOLLOW-UPS PTQ versus QAT trade-offs. Why are outliers a problem, they stretch the quantization range. What is mixed precision, keep sensitive layers higher precision. How do you measure the accuracy hit, evaluate on held-out tasks.

ONE CONCRETE EXAMPLE A 13B model in FP16 needs about 26 GB and will not fit a 24 GB GPU. Quantized to INT4 it drops near 7 to 8 GB and runs comfortably, with a small, measurable accuracy decrease that is acceptable for the use case.