CalorieMe: An Image-based Calorie Estimator System

In recent years, with the growing interest in healthy eating, various food photo recognition applications for tracking meals have emerged. However, some of these applications still require human intervention for calorie estimation, such as manual input or consultation with a nutrition expert. Furthermore, even automated systems often have limitations in food category recognition, or they demand multiple viewpoints for accurate results. Meanwhile, advancements in image recognition have been substantial, thanks to the advent of Convolutional Neural Networks (CNN). CNNs have significantly improved the accuracy of various image recognition tasks, including classification and object detection. This paper presents a comprehensive solution for estimating the calories in food photos containing multiple ingredients. The proposed method employs two deep learning models: one to detect ingredients and their respective locations, and another to segment the ingredients and measure their portion sizes. Moreover, the proposed method incorporates a reference object to enhance the precision of portion size measurement. Finally, the proposed model compares the food type and portion size against a dataset of food types and their corresponding calorific values per standard serving, thus estimating the total calorie count. In this study, the two-model methodology resulted in a 7% improvement in pixel accuracy and a 23% improvement in mean Intersection Over Union (mIOU) for recognition and segmentation tasks, respectively, compared to the latest state-of-the-art approach, which employed Deeplabv3+ exclusively on the same dataset.