Learning Neural Networks with Ranking-Based Losses for Action Retrieval

We consider the problem of learning image/video retrieval using a neural network based approach that optimizes the ROC loss function. Neural network is one of the most widely used techniques in computer vision. Standard neural network uses simple loss functions, such as the softmax loss or hinge loss over labels. Such loss functions are suitable for standard classification problems where the performance is measured by the overall accuracy. For image/video retrieval, the performance is usually measured by some ranking-based loss that is not well captured by the softmax loss or hinge loss. In this paper, we develop a learning approach that incorporates the ranking-based loss function in neural network. We apply our approach in the problem of action retrieval in static images and videos. The experimental results show that our proposed approach outperforms standard neural networks trained with softmax loss as well as an SVM-based approach that also optimizes the ROC loss function.