Action Recognition with Temporal Relationships

Action recognition is an important component in human-machine interactive systems and video analysis. Besides low-level actions, temporal relationships are also important for many actions, which are not fully studied for recognizing actions. We model the temporal structure of low-level actions based on dense trajectory groups. Trajectory groups are a higher level and more meaningful representation of actions than raw individual trajectories. Based on the temporal ordering of trajectory groups, we describe the temporal structure using Allen's temporal relations in a discriminative manner, and combine it with a generative model using bag-of-words. The simple idea behind the model is to extract mid-level features from domain-independent dense trajectories and classify the actions by exploring the temporal structure among them based on a set of Allen's relations. We compare the proposed approach with bag-of-words representation using public datasets, and the results show that our approach improves recognition accuracy.