Deep Reinforcement Learning for Video Prediction

This paper introduces a hybrid video prediction scheme that combines the classic parametric overlapped block motion compensation (POBMC) technique with neural networks. Most learning-based video prediction methods rely on a black-box-like model for either direct generation of future video frames or estimation of a dense motion field. The model complexity often increases drastically with frame resolution. Departing from pure black-box approaches, this paper leverages the theoretically-grounded POBMC in a reinforcement learning framework to estimate a sparse motion field for future frame warping. Two neural networks are trained to identify critical points in the motion field for motion estimation. We train our model on 10k unlabeled frames in KITTI dataset and achieve the state-of-the-art SSIM score of 0.923 on CaltechPed and an average SSIM scroe of 0.856 on Common Intermediate Format (CIF) standard sequences.