A Hybrid Architecture for Semantic Image Similarity Learning

Differentiating between similar image inputs has always been one of the key tasks in machine learning. Inspired by the recent progress in the area of natural language processing, we introduce the image similarity learning model that considers the semantic scene similarity in its decision process. The architecture of the model is organized in a way to consider the similarity in the feature vectors of the images, as well as the semantic similarity in their generated captions, which are later combined to reach a more accurate result. We use Siamese-like network structure for parallel image processing and receiving the accurate results. Our model confirmed to improve the accuracy of a standard convolutional neural network and was validated on INRIA Holidays Dataset.