Adapting a 2D CNN for video action recognition
extending image models to temporal data.
run the 2D CNN per frame, pool features over time, optionally add two-stream or 3D conv.
assuming a single frame suffices or ignoring temporal information entirely.
What's really being asked
The interviewer checks that you can leverage transfer learning from ImageNet while reasoning about the new temporal dimension that video adds. Actions are defined by motion, so a single frame is usually insufficient.
The full answer
Start by running the pretrained 2D CNN on each sampled frame to get a per-frame feature vector, reusing ImageNet weights as initialization. The simple aggregation method is temporal pooling: average or max the per-frame features across the clip to form one descriptor, then attach a classifier. Note this discards order, so stronger methods feed the frame features into an LSTM or temporal attention, or use a two-stream design adding an optical-flow branch for motion, or inflate the 2D kernels into 3D convolutions as in I3D to learn spatiotemporal filters directly.
The mistakes people make
Classifying from one frame and assuming appearance equals action, which fails for actions like opening versus closing a door. Or concatenating all frames into channels, which is brittle to clip length. Or jumping straight to a giant 3D network without mentioning the cheap per-frame plus pooling baseline.
What usually comes next
Why does average pooling lose temporal order and when does that matter. How does a two-stream network use optical flow. How can you inflate 2D weights to initialize a 3D network. How do you sample frames, uniformly or densely, and how does clip length affect cost.
A concrete example
For recognizing a tennis swing, you sample 16 frames spread across the clip, push each through ResNet-50 to get a 2048-dimensional feature vector, average them into one descriptor, and train a linear head on top. This baseline works well for appearance-heavy actions where context already reveals the activity. For motion-dependent classes such as sitting down versus standing up, you then add an optical-flow stream that captures direction of movement, or replace pooling with an LSTM or temporal attention so the order of frames informs the prediction, noticeably improving accuracy.
Interview question
What is the main limitation of average-pooling per-frame CNN features for action recognition?
- a.It increases the parameter count beyond a 3D CNN
- b.It only works for grayscale video input
- c.It cannot reuse ImageNet pretrained weights
- d.It discards the temporal order of framesCorrect
Why? this is the answer
Averaging is order-invariant, so it loses temporal sequence needed for direction-dependent actions. It is parameter-free, reuses pretrained weights, and works on color video, making the other options false.
Just read this? Test yourself on what you have been reading.
Read the original → arxiv.org
- #video
- #action-recognition
- #temporal-pooling
- #transfer-learning
- #cnn
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles