Cross-Modal Task Verification via Hypergraph-based Sequential Matching

Cross-Modal Task Verification (CMTV) assesses whether a procedural task is executed accurately according to language-based rules, presenting challenges due to its multi-modal and chronological nature. Existing methods using graph or neuro-symbolic approaches face two issues: (1) Conventional methods only model linear sequential relationships among intra-modal nodes, ignoring implicit relationships between non-neighboring nodes. (2) Directed graphs model pairwise cross-modal relationships but overlook cases where a step corresponds to multiple video segments. To address these issues, we propose Hypergraph-based Sequential Matching (HSM) with two components: (1) Temporal Complementary Hypergraph Module (TCHM), a hierarchical sequential hyperedge construction method that focuses on both sequential connections and implicit relationships across nodes. (2) Step-wise Hypergraph Modeling (SHM), a novel hypergraph-based alignment mechanism that better aligns an action description with multiple video segments, improving task verification accuracy. We evaluate HSM on EgoTV and CTV datasets, demonstrating its superiority over state-of-the-art methods.