Semantic Features for Food Image Recognition with Geo-Constraints

Food-related photos have become increasingly very popular, due to social networks, food recommendation and dietary assessment systems. Reliable annotation is essential in those systems, but user-contributed tags are often non-informative and inconsistent, and unconstrained automatic food recognition still has relatively low accuracy. Most works focus on exploiting only the visual content while ignoring the context. In this paper, we improve the food image recognition with using two novel components two kinds of context. Firstly, different from the conventional approach representing image in a visual feature space the visual features, we try to represent the images in the a semantic space (also called semantic simplex), where we model aiming at modeling more context information between each categories. Secondly, we explore leveraging leverage the geographic context of the user and information about geolocation and information about restaurants to simplify the classification problem. Thus, We propose a food recognition framework for the food recognition based on these two kinds of context, based on including semantic features learning and location-adaptive classification. We collected a restaurant-oriented food dataset with food images, dish tags and restaurant-level information, such as the menu and geographic location. Experiments on this dataset show that exploiting geolocation improves around 30% the recognition performance, and the semantic feature has a gain of 3%-10% to the other visual features.