Toward Low-Latency Cross-Modal Communication: A Flexible Prediction Scheme

To ensure the users' immersive experience in crossmodal communication, overcoming the end-to-end (E2E) latency through prediction has attracted attention and shown its superiority. However, existing prediction schemes encounter formidable challenges in the presence of multi-modal signals, primarily to adapt and satisfy the prediction requirements of diverse multi-modal services, as well as to fully exploit and effectively utilize the correlation features of multi-modal signals for precise prediction. To this end, this work presents a flexible prediction scheme for low-latency cross-modal communication. Specifically, we first propose an adaptive prediction-aware crossmodal communication framework, which reduces the delay by predicting and transmitting the future multi-modal signals in advance, and flexibly adjusts the prediction horizon to satisfy the prediction accuracy of different multi-modal services. Next, we design an information gain-assisted graph attention (IGGA) method for cross-modal signal prediction, which leverages the graph attention block to extract the intra-modal, inter-modal spatial and temporal correlation features, and effectively optimize and utilize these features with the information gain (IG), thereby facilitating precise cross-modal signal prediction. Finally, numerical experiments conducted on a self-built dataset, a public dataset, and a multi-modal acupuncture platform demonstrate the superiority of the proposed scheme in low-latency cross-modal communication.