Dual Spatial-Temporal Feature Pyramid With Decoupled Temporal Mining for Video Semantic Segmentation

Semantic segmentation plays a pivotal role in environmental perception for autonomous driving. Video semantic segmentation (VSS) further takes temporal information into consideration for better scene parsing and temporal consistency. Previous research on VSS is mostly dedicated to developing new techniques (e.g. optical flows, attention) to better mine temporal information. In this work, we contribute from a different angle by efficiently incorporating multi-scale temporal information. The dual spatial-temporal feature pyramid is proposed to enable the direct enhancement of multi-scale features for target frames and unlash the design of temporal information mining modules. It contains a spatial feature pyramid from a target frame and a spatial-temporal feature pyramid from multiple reference frames. Building on the dual feature pyramid, we further propose to decouple motional contexts and static contexts to fully leverage temporal information. Specifically, multi-scale motional contexts are mined with the introduced dedicated module and static contexts are enhanced by making temporally fused category-level representations interact with the target frame feature. The final segmentation maps are obtained by regarding the enhanced category-level representations as powerful feature classifiers to classify the target frame feature of rich motional contexts. Experimental results on two popular VSS benchmarks demonstrate that the proposed method with decent parameter and inference efficiency clearly outperforms previous advanced methods.