This note argues that a 3D convolutional network and an LSTM work far better together than either does alone when actions unfold over time. It starts from the limits of a plain CNN on long video, then narrows the gap to the 3D-CNN, which reads motion inside a short clip but loses the wider context. The proposed fix splits a video into clips, extracts spatio-temporal features with C3D or I3D, and passes the sequence to an LSTM. A simple mental picture follows, the eyes and the brain, then the advantages. The note closes with the two parts of the model, examples from papers, and open comparisons such as ST-GCN and a Transformer replacing the LSTM.
3D-CNN + LSTM: A Powerful Pair for Action Recognition
The problem with a plain CNN
- A CNN only suits still images or short videos
- For long videos and complex actions, a CNN isn't enough
Limitations of 3D-CNN
- Strong at: Capturing local motion and understanding adjacent frames within a short clip
- Weak at: Long videos and actions that unfold over time (climbing, playing piano, martial arts, etc.) → loses the overall context
- Solution: A "long-term memory" is needed → LSTM
How to combine 3D-CNN + LSTM
Pipeline:
- Split the video into small clips (a few dozen frames each)
- Use a 3D-CNN (C3D / I3D) to extract spatio-temporal features from each clip
- Feed the sequence of feature vectors into an LSTM → understand the sequence of movements from start to finish
Mental picture:
- 👁 The 3D-CNN is the eyes → sees each segment clearly
- 🧠 The LSTM is the brain → strings the segments together → understands the whole action
Advantages of the combination
- ✅ Understands both short and long movements
- ✅ Decodes complex, structured actions
- ✅ Suits long videos, or videos with several consecutive stages (e.g., "sit down then stand up", "jump then turn around")
Conclusion
- 3D-CNN × LSTM = a very strong combo for hard action recognition tasks
- Combines the CNN's ability to "see" with the RNN's ability to "remember"
- The model is both more accurate and far more general
- Used in many top papers from 2016 to the present:
- "Convolutional Two-Stream Network Fusion for Video Action Recognition"
- "Deep Temporal Linear Encoding"
Comparison and Future Directions
- Comparison: 3D-CNN + LSTM vs ST-GCN for keypoint-based recognition
- Alternative: Transformer in place of LSTM
See also: RNN And LSTM · Pose Estimation · Vision Transformers