Image scoring: Patch based CNN model for small or medium dataset

Image scoring focuses on visual classification or regression which assigns each image a category or precise score. As deep learning is becoming more and more popular, large manually labeled data sets are required. This makes the task time-consuming, and is difficult for small or medium dataset. In this paper, we give a simple yet effective oversampling technique that considers both entire image and local patches. Oversampling is a well-known trick in deep learning, while in this paper it mainly focuses on the small-size patches instead of some large-size patches that cover the entire object of the image. We first crop an image into many small-size patches that can augment the initial dataset and also partially decrease the over-fitting in the training. The initial dataset and expanded dataset can separately be seen as global and local information for each image. Based on the expanded dataset, we can train standard Convolutional Neural Network (CNN) and patch based CNN (patCNN). In order to integrate the global and local information, we further combine two models to get better performance, called comCNN. The experimental results show the effectiveness of patCNN and comCNN compared with the state-of-the-art methods.