Evidential Multimodal Fusion Network for Trusted Pedestrian Crossing Intent Prediction

Accurate prediction of pedestrians’ behavior poses formidable challenges for autonomous vehicles in urban environments. Multimodal data, such as pedestrians’ motion data, context images, and ego vehicle speed, offer complementary and comprehensive information that can significantly enhance prediction performance. However, the previous methods, despite yielding promising results, integrate different modalities to form a uniform representation, which falls short of fully exploiting the heterogeneity and complementarity of all modalities. Besides, the uncertainty inherent in predictions is also a major concern for safety-critical systems such as autonomous vehicles. In light of the above concerns, this study puts forward a novel evidential multimodal fusion network called EMFNet, which leverages multimodal data and evidence fusion techniques for trusted pedestrian crossing intention prediction. Specifically, modal-specific embedding is first developed to project raw multimodal data into high-dimensional space by considering the unique properties of each modality. Then, we devise intra-modal and cross-modal feature learning to capture the temporal correlation within each modality and the interaction across different modalities, respectively. By doing so, modal-invariant features and modal-specific features can be effectively extracted. Subsequently, we introduce the Dempster–Shafer’s evidence theory (DST) to amalgamate evidence associated with different modalities, thus allowing the model to estimate the uncertainty and achieve trusted crossing prediction. Finally, we design an adaptive multiloss function that can effectively supervise the learning of modal-invariant and model-specific and facilitate the evidence fusion process. Experimental evaluation on real-world benchmark datasets demonstrates the improved prediction performance and enhanced reliability of the proposed method.