MCFM: Mutual Cross Fusion Module for Intermediate Fusion-Based Action Segmentation

This paper presents an action segmentation method utilizing multiple features on the basis of a novel intermediate fusion module, named Mutual Cross Fusion Module (MCFM). The proposed method analyzes multiple features on each feature’s classifier stream. MCFM recalibrates the target feature in the middle of the classifier stream from the knowledge of the other features in contrast to the existing module of the previous method, which utilizes a joint representation learned from all features for the recalibration of the target feature. MCFM integrates the knowledge of multiple features without biassing the knowledge toward one of the multiple features. We compare the proposed method with the state-of-the-art methods on two public datasets: GTEA and 50Salads. The proposed method outperforms the state-of-the-art methods in terms of frame-wise accuracy, edit distance, and F1-score by 2.0, 1.6, and 2.9 points, respectively.