Appearance, context and co-occurrence ensembles for identity recognition in personal photo collections

While modern research in face recognition has focused on new feature representations, alternate learning methods for fusion of features, most have ignored the issue of unmodeled correlations in face data when combining diverse features such as similar visual regions, attributes, appearance frequency, etc. Conventional wisdom is that by using sufficient data and machine, one can learn the systematic correlations and use the data to form a more robust basis for core recognition tasks like verification, identification, and clustering. This however, takes large amounts of training data which is not really available for personal consumer photo collections. We address the fusion/correlation issue differently by proposing an ensemble-based approach that is built on different information sources such as facial appearance, visual context, and social (or co-occurrence) information of samples in a dataset, to provide higher classification accuracy for face recognition in consumer photo collections. To evaluate the utility of our ensembles and simultaneously generate stronger generic features, we perform two experiments — (i) a verification experiment on the standard unconstrained LFW (Labeled Faces in the Wild) dataset where by using an ensemble of appearance related features we report comparable results with recently reported state-of-the-art results and 2.9% better classification accuracy than the previous best method, and(ii) experiment on the Gallagher personal photo collection where we demonstrate at least 17% relative performance gain using visual context and social co-occurrence ensembles.