Depthwise separable convolution cost savings

WHAT THIS TESTS Whether you can decompose a convolution's cost and articulate the assumption that makes the cheaper version work, the basis of MobileNet and Xception.

A GOOD ANSWER COVERS A standard three by three convolution with M input channels and N output channels over a feature map of spatial size D by D costs roughly three times three times M times N times D times D multiply-adds and three times three times M times N parameters, because every output channel filters over all input channels and a spatial window jointly. A depthwise separable convolution splits this into two steps. The depthwise step applies a single three by three filter to each input channel independently, costing three times three times M times D times D. The pointwise step is a one by one convolution that combines the M channels into N outputs, costing M times N times D times D. The ratio of separable to standard cost is approximately one over N plus one over nine, so for typical channel counts you save roughly eight to nine times in both FLOPs and parameters.

COMMON WRONG ANSWERS Forgetting the pointwise step, which alone would prevent any cross-channel mixing. Miscomputing by treating depthwise as if it still spans all channels. Claiming there is no accuracy tradeoff; there is usually a small drop.

LIKELY FOLLOW-UPS What assumption justifies the factorization. Why does MobileNet add width and resolution multipliers. How does Xception interpret this. When does the small accuracy loss matter.

ONE CONCRETE EXAMPLE With M equals N equals two-fifty-six and a three by three kernel, the standard conv does three times three times two-fifty-six times two-fifty-six work per spatial location, while the separable version does three times three times two-fifty-six for depthwise plus two-fifty-six times two-fifty-six for pointwise. The ratio is about one ninth plus one over two-fifty-six, roughly an eight to nine times reduction, which is why MobileNets run on phones. The underlying hypothesis is that spatial filtering and cross-channel recombination are largely independent and can be learned in two cheaper steps rather than one expensive joint operation.