Feature Analysis in Camera and Mm-Wave Radar Fusion for Soccer Player Action Recognition

Player tracking systems in soccer provide valuable insights for broadcasters, coaches and teams. Action recognition enhances these systems by providing player statistics in addition to player tracking data. This study investigates which features are suitable for soccer player action recognition when fusing radar and camera data. The created dataset consists of four movement classes: walk, jog, dribble walk, and dribble jog. The data was obtained using mm-Wave industrial radar and a smartphone camera. PCA, t-SNE and random forest feature importance analysis is performed and it is concluded that the fusion of radar and camera data better separates the classes than only using any sensor in isolation for the soccer player action recognition scenario. The histogram of oriented gradients of both the full frame and the region of interest are determined to be the best camera features, whereas the mel-spectrogram and mel-coefficients are the best radar features according to the random forest feature importance analysis. For the radar-camera fusion the HOG for the full frame along with mel-spectrograms and some simple radar features were determined to be the best features and using them achieved a high classification accuracy.