Query-based retrieval of complex activities using “strings of motion-words”

Analysis of activities in low-resolution videos or far fields is a research challenge which has not received much attention. In this application scenario, it is often the case that the motion of the objects in the scene is the only low-level information available, other features like shape or color being unreliable. Also, typical videos consist of interactions of multiple objects which pose a major vision challenge. This paper proposes a method to classify activities of multiple interacting objects in low-resolution video by modeling them through a set of novel discriminative features which rely only on the object tracks. The noisy tracks of multiple objects are transformed into a feature space that encapsulates the individual characteristics of the tracks, as well as their interactions. Based on this feature vector, we propose an energy minimization approach to optimally divide the object tracks and their relative distances into meaningful partitions, called "strings of motion-words". Distances between activities can now be computed by comparing two strings. Complex activities can be broken up into strings and comparisons done separately for each object or for their interactions. We test the efficacy of our approach to search all the instances of a given query in multiple real-life video datasets.