MAPLE: Modality-Agnostic Prototype Learning for Egocentric Action Recognition

In this paper, we address the novel task of egocentric modality generalization action recognition, which aims to learn a unified discrete representation from paired multimodal egocentric action data during pre-training. This approach enables cross-modal zero-shot generalization in downstream tasks, where the modalities available during inference and training are disjoint. While recent efforts have focused on aligning instance-level or temporal features to reduce feature distribution discrepancies across modalities, they have overlooked the inherent structural categorization within action data. To address this limitation, we propose Modal-Agnostic Prototype Learning (MAPLE), a framework that leverages a prototype memory bank to capture categorical structures. This is further enhanced by a robust semantic disentanglement module and a moment aggregation mechanism, enabling semantically similar behaviors to cluster more closely in the latent space and promoting robust cross-modal generalization. Extensive experiments on the Ego4D and WEAR datasets demonstrate that MAPLE significantly outperforms existing approaches, marking a substantial advancement in the field of egocentric action recognition.