Handling outlier activations in INT8 quantization

WHAT THIS TESTS This probes a real failure mode of low-bit quantization: a small number of high-magnitude activation outliers can wreck INT8 accuracy, and whether you know a targeted mixed-precision remedy rather than abandoning quantization.

A GOOD ANSWER COVERS INT8 quantization maps a tensor's floating-point range onto 256 integer levels using a scale factor. If a few activation values are enormous compared to the rest, the scale must stretch to cover them, so the vast majority of normal-magnitude values get squeezed into a tiny portion of the integer range and lose precision, degrading accuracy. To find the culprits, run representative calibration data through the model and collect per-channel activation statistics, the max absolute value, distribution, or high percentiles per feature dimension. Outlier channels show up as a small set with magnitudes far above the rest, often concentrated in specific feature dimensions and growing in larger models. The mitigation is mixed precision: keep the identified outlier channels in higher precision such as FP16 while quantizing the remaining well-behaved channels to INT8, performing a decomposition so most matrix multiplication runs in fast INT8 and only the outlier portion runs in FP16. An alternative is to migrate the scale difficulty from activations into weights via a smoothing transform so both quantize cleanly. Either way you preserve most of the speed and memory benefit while recovering accuracy.

COMMON WRONG ANSWERS Blaming the weights rather than activations; abandoning quantization entirely; reducing to even lower bits expecting improvement; quantizing uniformly with no per-channel analysis; ignoring calibration data.

LIKELY FOLLOW-UPS Why do outliers concentrate in certain channels? How does per-channel versus per-tensor scaling help? What is activation smoothing? How much speed do you sacrifice keeping some channels in FP16?

ONE CONCRETE EXAMPLE Profiling activations on calibration prompts, you find that a handful of feature dimensions carry values an order of magnitude larger than the rest, forcing a coarse INT8 scale. You decompose the matrix multiply so those outlier dimensions stay in FP16 while the other dimensions run in INT8. Accuracy on the key metric returns near baseline, and most of the throughput and memory savings are retained.