Do I Know You? A Two-Stage Framework for Novelty Detection

In this work, we address the problem of novelty detection in the context of image classification, where the goal is to classify whether a query belongs to one of the classes seen during training or to a novel class. Given a network trained on a set of training classes, we utilize the activations from the final fully connected layer in a two-stage framework to determine whether a given query is seen or novel. In the first stage, for a given query, analyzing the top retrieved samples gives us a set of probable categories for that query. For the second stage, a comparator network is designed which can compare features from two samples and classify them into similar or dissimilar pairs. Prototype exemplars from each of the probable categories computed in the first stage are compared with the query feature to determine whether they come from the same or different category. The scores from both the stages are fused using a simple yet effective fuzzy aggregation operator to give a final decision on whether the given query is from a seen or novel class. Extensive experiments on three benchmark databases, MNIST, HASYv2, and Fashion-MNIST demonstrate the effectiveness of the proposed framework.