Audio Visual Graph Attention Networks for Event Detection in Sports Video

Interest in event detection for sports video analytics has been increasing because it enables teams to make game plans and can help to reduce the huge working costs of broadcast production. Sports event detection is a challenging task due to it requiring both audio and video features. Several recent methods have modified the transformer model to enable extraction of multi-modal features. These methods input the combined audio and video to the multi-modal transformer and therefore take into account multi-modal interactions. However, the transformer-based method does not focus on significant interactions from the modal interactions, which include ones with emphasis on audio, emphasis on video, or consideration of both audio and video, because it does not pay attention to specific interactions but rather all interactions at once. In this paper, we propose the Audio Visual Graph Attention Networks (AVGAT) model that prepares a different graphical structure for each modal interaction. Consequently, our model can pay attention to the significant modal interactions. Experiments on datasets of actual wheelchair-rugby games showed that AVGAT outperformed the transformer-based model. In addition, an ablation study showed that each component of our model empirically functions well at detecting individual event labels.