Fusing Images and Ontologies for Situation Representation in Knowledge Graphs

In Smart City applications, urban mobility involves complex interactions between traffic infrastructure, diverse road users, and physical environment. This paper addresses the limitations of conventional scene modeling methods that often fail to capture the varied and volatile nature of urban road scenes, particularly in representing the dynamic situations that unfold within them. Inaccurate situation representation hinders precise depiction of scene evolution, limiting our ability to understand and respond effectively to complex urban situations. This paper addresses these challenges by presenting the novel concept of the Context-Aware Scene Graph (CSG) for representing situations used in reasoning applications for enhancing safety and efficiency in urban environments, particularly for bicycle riders. CSG integrates multi-modal data, including ontological knowledge, sensor data, and images, to provide a comprehensive representation of urban road situations, enabling informed decision-making. This paper also validates the effectiveness of the proposed approach using real- world IoT datasets and camera images, with a focus on the bicycle dooring use case. The results outperform existing scene modeling methods by accurately representing situations, including those previously overlooked. The approach also ensures consistent representation, completeness, and captures transitions between situations, including causal relations. These findings highlight our approach's effectiveness in improving road safety, efficiency, and urban life quality through enhanced scene understanding.