FACT++: Multi-Stage Hand Keypoint Fusion for Enhanced Egocentric Action Segmentation

Temporal action segmentation in egocentric videos is vital for assistive technologies and activity recognition. Existing methods often neglect hand-keypoint data, which is crucial for capturing subtle hand-object interactions. We enhance the state-of-the-art FACT framework by integrating hand keypoints extracted from the GTEA dataset using MediaPipe. These keypoints are fused at multiple network stages via a weighted strategy, improving the modeling of fine-grained actions. Our approach boosts segmentation accuracy, achieving an average of 86.3 % across test splits and a peak of 88.6 % on split 1, highlighting the value of hand keypoint data while preserving efficiency.