Learning Neural Bag-of-Features for Large-Scale Image Retrieval

In this paper, the well-known bag-of-features (BoFs) model is generalized and formulated as a neural network that is composed of three layers: 1) a radial basis function (RBF) layer; 2) an accumulation layer; and 3) a fully connected layer. This formulation allows for decoupling the representation size from the number of used codewords, as well as for better modeling the feature distribution using a separate trainable scaling parameter for each RBF neuron. The resulting network, called retrieval-oriented neural BoF (RN-BoF), is trained using regular back propagation and allows for fast extraction of compact image representations. It is demonstrated that the RN-BoF model is capable of: 1) increasing the object encoding and retrieval speed; 2) reducing the extracted representation size; and 3) increasing the retrieval precision. A symmetry-aware spatial segmentation technique is also proposed to further reduce the encoding time and the storage requirements and allows the method to efficiently scale to large datasets. The proposed method is evaluated and compared to other state-of-the-art techniques using five different image datasets, including the large-scale YouTube Faces database.