A Semantic Inpainting Framework for Distributed Cross-Modal Integrated Sensing and Communication

This paper proposes a semantic inpainting framework for integrated cross-modal sensing and communication. Existing research on integrated sensing and communication (ISAC) primarily focuses on integrating multiple modalities in data space to enhance sensing accuracy. In contrast, inspired by the joint embedding predictive architecture (JEPA) principle, the proposed framework focuses on inpainting and reconstruction in the semantic space. The ability to predict and reconstruct semantics rather than raw data enables cross-modal prediction of modalities without collecting data. Additionally, it can compensate for missing modalities caused by sensor or communication failures. Using a multimodal dataset comprising images (from cameras) and RF modality (Wi-Fi CSI), we demonstrate the feasibility of cross-modal semantic prediction through an experimental image reconstruction application. Furthermore, the proposed method demonstrated approximately a 20% improvement in image reconstruction quality compared to the single-modal JEPA.