A New Video Dataset for Recognizing Intake Gestures in a Cafeteria Setting

Intake gesture detection plays an important role in dietary monitoring systems to help individuals objectively quantify their eating behavior. Intake gestures include motions related to consumption, such as ingestion of a bite (food) and drink (liquid). Video provides an automatic and non-contact way to detect gestures. Previous studies have evaluated video-based detectors in laboratories where participants were invited to consume meals containing a limited number of food and beverage choices. Because of this, previous datasets have limited variability in food and beverage appearances, and types of intake tools and gestures. Cameras were positioned conspicuously close to participants. To address these limitations, we describe a new video dataset that was collected by recording the meals of 264 participants in a real restaurant. Participants consumed 374 different types of foods and beverages. The camera was installed on the ceiling of the restaurant to minimize the effect of an experimental observer. In this study, we present the details of collecting and labeling the data for video-based research. To explore the feasibility of detecting intake gestures using our dataset and establish baseline performance, we implemented three state-of-art models, namely CNN-LSTM, SlowFast and X3D-L. Results indicate that the best F1 scores were 0.899 and 0.778 on bite and drink gesture detection respectively.