A Novel Double-Layer Framework for Action Prediction

Action prediction aims to infer the category of an action before it is fully executed. It is a challenging task since neither sufficient discriminative information nor the definite progress state of action can be obtained in an incomplete video. In this paper, we propose a novel double-layer learning framework for predicting the category of action from partial observations. Particularly, in the first layer of the framework, an unsupervised semantic reasoning method is presented for exploiting semantic information of an input incomplete video as well as inferring the future semantic information using the prior knowledge provided by training full videos. In the second layer of the framework, a discriminative action prediction model introduces a latent variable to indicate the progress state of the input video and captures the relationship among the actions, video observations, the semantic information, and the latent progress state for predicting the action label of the input video. Extensive experimental results on UT-I #1, UT-I #2, and UCF Sports datasets demonstrate the superiority of our method in predicting actions at the early stage.