Unsupervised Learning of Long-Term Motion Dynamics for Videos


We present an unsupervised representation learning approach that compactly encodes the motion dependencies in videos. Given a pair of images from a video clip, our framework learns to predict the long-term 3D motions. To reduce the complexity of the learning framework, we propose to describe the motion as a sequence of atomic 3D flows computed with RGB-D modality. We use a Recurrent Neural Network based Encoder-Decoder framework to predict these sequences of flows. We argue that in order for the decoder to reconstruct these sequences, the encoder must learn a robust video representation that captures long-term motion dependencies and spatial-temporal relations. We demonstrate the effectiveness of our learned temporal representations on activity classification across multiple modalities and datasets such as NTU RGB+D and MSR Daily Activity 3D. Our framework is generic to any input modality, i.e., RGB, depth, and RGB-D videos.

11 Figures and Tables

Cite this paper

@article{Luo2017UnsupervisedLO, title={Unsupervised Learning of Long-Term Motion Dynamics for Videos}, author={Zelun Luo and Boya Peng and De-An Huang and Alexandre Alahi and Li Fei-Fei}, journal={CoRR}, year={2017}, volume={abs/1701.01821} }