Adapting a 2D CNN for video action recognition

WHAT THIS TESTS The interviewer checks that you can leverage transfer learning from ImageNet while reasoning about the new temporal dimension that video adds. Actions are defined by motion, so a single frame is usually insufficient.

A GOOD ANSWER COVERS Start by running the pretrained 2D CNN on each sampled frame to get a per-frame feature vector, reusing ImageNet weights as initialization. The simple aggregation method is temporal pooling: average or max the per-frame features across the clip to form one descriptor, then attach a classifier. Note this discards order, so stronger methods feed the frame features into an LSTM or temporal attention, or use a two-stream design adding an optical-flow branch for motion, or inflate the 2D kernels into 3D convolutions as in I3D to learn spatiotemporal filters directly.

COMMON WRONG ANSWERS Classifying from one frame and assuming appearance equals action, which fails for actions like opening versus closing a door. Or concatenating all frames into channels, which is brittle to clip length. Or jumping straight to a giant 3D network without mentioning the cheap per-frame plus pooling baseline.

LIKELY FOLLOW-UPS Why does average pooling lose temporal order and when does that matter. How does a two-stream network use optical flow. How can you inflate 2D weights to initialize a 3D network. How do you sample frames, uniformly or densely, and how does clip length affect cost.

ONE CONCRETE EXAMPLE For recognizing a tennis swing, you sample 16 frames spread across the clip, push each through ResNet-50 to get a 2048-dimensional feature vector, average them into one descriptor, and train a linear head on top. This baseline works well for appearance-heavy actions where context already reveals the activity. For motion-dependent classes such as sitting down versus standing up, you then add an optical-flow stream that captures direction of movement, or replace pooling with an LSTM or temporal attention so the order of frames informs the prediction, noticeably improving accuracy.