EGGNOG: A Continuous, Multi-modal Data Set of Naturally Occurring Gestures with Ground Truth Labels

People communicate through words and gestures,but current voice-based computer interfaces such as Siri exploitonly words. This is a shame: human-computer interfaces wouldbe natural if they incorporated gestures as well as words. To support this goal, we present a new dataset of naturally occurring gestures made by people working collaboratively on blocks world tasks. The dataset, called EGGNOG, contains over 8 hours ofRGB video, depth video, and Kinect v2 body position data of 40subjects. The data has been semi-automatically segmented into 24,503 movements, each of which has been labeled according to (1) its physical motion and (2) the intent of the participant.We believe this dataset will stimulate research into natural and gestural human-computer interfaces.