Semi-Supervised Cross-Modality Action Recognition by Latent Tensor Transfer Learning

Microsoft's Kinect sensors are receiving an increasing amount of interests by security researchers since they are cost-effective and can provide both visual and depth modality data at the same time. Unfortunately, depth or RGB modalities are unavailable in training or testing procedures in some realistic scenarios. Therefore, we explore a new problem focusing on the arbitrary absence of modality, which is completely different from the conventional action recognition. The new problem in action recognition aims to deal with cross modality data (e.g., RGB training and depth testing data), “missing” modality data (e.g., RGB training and RGB-D test data), and single-modality data (e.g., RGB/depth in both phases). Accordingly, our method aims to borrow some information (e.g., correlation between two modalities) from the well-established RGB-D dataset and apply it to the existing dataset to recover some latent information to improve the performance of recognition. For instance, a cross-modality regularizer is used to preserve the correlation of RGB and depth modalities. The “missing” knowledge is considered as latent information, which is recovered by low-rank learning in our model. In the real world, the target data are usually sparsely labeled or completely unlabeled; however, we could exploit the pseudolabels of the target as prior knowledge for “supervised” learning in the target domain. Accordingly, we propose a semi-supervised model for transfer learning. The experiments on three widely used RGB-D action datasets show that our method performs better than that of the state-of-the-art transfer learning methods in most cases in terms of accuracy and time efficiency.