Learning a joint discriminative-generative model for action recognition

An action consists of a sequence of instantaneous motion patterns whose temporal ordering contains critical information especially for distinguishing fine-grained action categories. However, existing action recognition methods are dominated by discriminative classifiers such as kernel machines or metric learning with Bag-of-Words (BoW) action representations. They ignore the temporal structures of actions in exchange for robustness against noise. Although such temporal structures can be modelled explicitly using dynamic generative models such as Hidden Markov Models (HMMs), these generative models are designed to maximise the likelihood of the data therefore providing no guarantee on suitability for discrimination required by action recognition. In this work, a novel approach is proposed to explore the best of both worlds by discriminatively learning a generative action model. Specifically, our approach is based on discriminative Fisher kernel learning which learns a dynamic generative model so that the distance between the log-likelihood gradients induced by two actions of the same class is minimised. We demonstrate the advantages of the proposed model over the state-of-the-art action recognition methods using two challenging benchmark datasets of complex actions.