Automatic labelling of touristic pictures using CNNs and metadata information

In this paper, we present a system that automatically recognizes pictures of landmarks in Singapore and combine the output with metadata information extracted from the picture if available in order to provide tourists with information about the recognized places. In detail, the system combines GPS information from the image metadata with a Convolutional Neural Network (CNN) based image recognition system to caption the image, where the GPS information is used to verify the caption given by the CNN, making the system more robust. For the training of the CNN, a total of ~67000 images were automatically downloaded from different search engines including Google, Flickr and Bing which were then filtered down to ~330 images per landmark to preserve the quality of the training dataset (e.g. removing repeated or confusing images), and then artificially augmented. After optimization, our CNN achieves a F1 score of 81% over a set of 6 different but very challenging locations, with each being a popular landmark in Singapore. With this combined system, the system can accurately tell the user where the image was taken and what the landmark in the image is. In addition, we created a database that provides background information for many landmarks (including, but not limited to the 6 landmarks used for training), such as their historic al or cultural significance. In order to make the system ac cessible and intuitive, it was integrated into a website where a Google Map is generated showing the results of the recognition and metadata information.