Core insight behind GPTQ and AWQ

WHAT THIS TESTS This advanced question checks whether you understand why INT4 is hard and the specific insight that makes it viable: weights are not equally important, and error can be compensated or steered away from what matters.

A GOOD ANSWER COVERS The failure of naive methods: round-to-nearest quantization to INT4 treats every weight independently and uniformly, so accumulated rounding error, amplified by activation outliers, severely degrades output. The shared insight of advanced methods is that not all weights contribute equally to the layer's output, so quantization should minimize the impact on outputs, not the per-weight rounding error. GPTQ frames quantization as minimizing the layer's output reconstruction error and quantizes weights greedily, one column or group at a time, while using approximate second-order curvature, a Hessian estimate from calibration activations, to update the not-yet-quantized weights so they compensate for the error introduced, à la optimal brain quantization. AWQ, activation-aware weight quantization, observes that protecting a small fraction of salient weight channels, those multiplied by large-magnitude activations, preserves most of the accuracy; it identifies these channels from activation statistics and applies per-channel scaling so the important weights are quantized with less relative error, without needing backprop. Both rely on a small calibration set to capture activation statistics rather than retraining.

COMMON WRONG ANSWERS Saying they simply round more carefully or use more bits secretly. Claiming all weights are treated uniformly. Confusing them with QAT, which retrains weights, these are post-training methods. Attributing the gain to better hardware.

LIKELY FOLLOW-UPS Why does AWQ scale rather than keep channels in FP16, scaling avoids mixed precision overhead. What does the Hessian capture in GPTQ, weight sensitivity. How big is the calibration set, small, hundreds of samples. How do these compare to SmoothQuant.

ONE CONCRETE EXAMPLE Naive INT4 on a 7B model collapses perplexity, while AWQ identifies roughly the top one percent of activation-aligned channels and scales them, recovering near-FP16 quality at one-quarter the memory using only a small calibration set.