Joint Statistical and Spatial Sparse Representation for Robust Image and Image-Set Classification

Recent image classification schemes, by learning deep features from large-scale dataset, have achieved the significantly better results comparing to classic feature-based approaches. However, there are still challenges in practice, such as classifying noisy image-set queries and training over limited-scale dataset. Instead of applying generic deep features, the model-based approaches can be more effective for robust image and image-set classification tasks, as we need various image priors to exploit the inter- and intra-set data variations while prevent over-fitting. In this work, we propose a novel joint statistical and spatial sparse representation, dubbed J3S, to model the image or image-set data, by exploiting both their local patch structures and global Gaussian distribution into Riemannian manifold. To the best of our knowledge, no work to date utilized both global statistics and local patch structures jointly via sparse representation. We propose to solve a co-regularized sparse coding problem based on the J3S model, by coupling the local and global representations using joint sparsity. The learned J3S models are used for robust image and image-set classification. Experiments show that the proposed J3S-based image classification scheme outperforms the popular or state-of-the-art competing methods.