Heterogeneous Stream Scheduling for Cross-Modal Transmission

Cross-modal communication is playing an increasingly important role in improving receivers' immersive experience. The main challenge lies in ensuring the heterogeneous requirements of the cross-modal stream. Especially, the discontinuity of the received haptic signals caused by the delay should be eliminated. Unfortunately, existing schemes study the cross-modal stream separately, which leads to that the haptic signal is distorted, or the audio-visual quality is reduced. To solve this problem fundamentally, we propose a joint transmission framework combining prediction and device-to-device (D2D) communication by taking advantage of the correlation of the haptic signals and the proximity feature of the receivers. Specifically, on the theoretical end, to completely eliminate the discontinuity, we propose a prediction mechanism by predicting and sending the future signals in advance. To compensate for the reliability loss brought by prediction, D2D links are efficiently established on the receivers' side. On the technical end, we first design a minimum resource (e.g., power) consumption search algorithm based on the binary search to obtain the optimal prediction horizon. Moreover, we develop a simple but efficient transmission mode selection algorithm based on the Hungarian algorithm. Experimental results demonstrate the advantages of our proposed scheme in saving the power consumption.