Hotel Recognition Using Object Ensembles

Hotel recognition from images is a key part of investigations of child sexual abuse and human trafficking, where victims are often photographed in hotels; identifying the place can be a key step in building cases against abusers and traffickers. Recent approaches to hotel recognition use a standard image retrieval pipeline – given a query image and a sufficiently large corpus of images with known locations, the query image’s location is inferred from the most similar images in the database. These approaches are based on extracting image representations from deep learning models trained specifically on the hotel recognition task using a metric learning loss that forces images from the same hotel have different representations, and images from different location to have different representations. This ‘whole image’ based approach, however, can be sub-optimal for hotel recognition, as images from the same hotel may look quite dissimilar (rooms in different configurations, rooms from before and after renovations, etc.), images from different hotels may look quite similar (hotels of the same chain or style), and, especially in real world investigations, the victim may occlude a very large portion of the image.To address these challenges, we propose an object-centric approach to the hotel recognition task. Specifically, we use ensemble learning, training separate models for each different object type. We then present a light weight approach to combining the decisions from different object models in the ensemble. At query time, features are extracted for each visible object in the query image from its respective object-specific model, and a final classification decision is made using the ensemble.We evaluate our approach on highly occluded images from the Hotel-ID to Combat Human Trafficking 2022 dataset, and present a comparison of our object-centric ensemble learning approach with more standard image-based approaches and discuss the properties of images where the object-centric approach achieves superior performance on the hotel-recognition task.