Camera Coach: Activity Recognition and Assessment Using Thermal and RGB Videos

Human Activity Recognition (HAR) has sparked a lot of attention because of its wide range of applications in sports, animation, simulation, and entertainment. There are many applications based on HAR, like healthcare monitoring, entertainment applications, and athletes' performance analysis. During the Corona pandemic, people took remote training as a safe and easy solution to do sports activities during the closure period, so there was a need to use smart systems to help people to train alone. In this paper, we expand our work on our dataset Multi-Modal Dataset of Sports (MMDOS); this dataset contains a variety of data forms from different sensors, including RGB videos, inertial motion data, depth, and thermal data, all synchronized with regard to the activities/exercises performed. The dataset consists of four workout exercises: free squats, shoulder press, push-ups, and lunges. The exercises are performed by 50 participants indoors at a fitness center. Professional trainers label the data with the type of activity and the participant's mistakes. Data are collected from different sensors, including Red-Green-Blue (RGB) videos, depth videos and skeleton data, inertial motion units (IMU) data, and thermal data; all are synchronized. From these modalities, we specifically used RGB, thermal videos, and 3-D skeleton of depth videos to recognize and asses the four activities. We built a smart Gym coaching model using classical machine learning (ML) and deep learning (DL) methods. The proposed system achieved an outstanding performance of 99% for activity recognition and an error of 0.75 out of 10 for activity assessment.