Camera-Based Meal Type and Weight Estimation in Self-Service Lunch Line Restaurants

Individuals, restaurant owners and health organizations are all interested in accurate information about food intake, but collecting the information in sufficiently automated way remains a practical challenge. In controlled environments, such as lunch line restaurants, the food intake can be estimated by measuring the portions upon purchase and separately monitoring the food waste, but even this requires often complicated setups such as repeated weighing of the plate after every meal component. In this work we explore the feasibility of using a combination of ceiling-mounted cameras and computer vision for estimating both the types and weights of individual food items the customers are taking in a lunch line restaurant. We describe the imaging system and weighing-based sensing for obtaining ground truth training data, and develop and evaluate deep learning models for the computer vision tasks. We demonstrate high accuracy especially in meal type identification and hence validate the feasibility of the approach. We release the annotated dataset for further development of improved methods.