Super-Sparse Learning in Similarity Spaces

In a growing number of applications, including computer vision, biometrics, text categorization and information retrieval, samples are often represented more naturally in terms of similarities between each other, rather than in an explicit feature vector space [1], [2]. Traditional machine-learning algorithms can still be used to learn over similarity-based representations; e.g., linear classification algorithms like Support Vector Machines (SVMs) [3], [4] can be trained in the space implicitly induced by the similarity measure (i.e., the kernel function) to learn nonlinear functions in input space. However, the main drawback of similarity-based techniques is their high computational complexity at test time, since computing their classification function often requires matching the input sample against a large set of reference prototypes, and evaluating such similarity measures is usually computationally demanding. Even SVMs, that induce sparsity in the number of required prototypes (the so-called support vectors, SVs) may not provide solutions that are sparse enough, as the number of prototypes (i.e., SVs) grows linearly with respect to the number of training samples [5], [6]. To reduce the number of reference prototypes, several state-of-the-art approaches select them from the training data, and then separately train the classification function using the reduced set of prototypes. However, decoupling these two steps may not effectively reduce the number of prototypes, without significantly affecting classification accuracy [2], [7], [8].