Vision-Assisted Cross-Modal Haptic Data Compression for Immersive Communication

Haptic data compression has gradually become a key issue for emerging immersive communications. However, it is challenging to achieve a trade-off between high perceptual quality and compression ratio in haptic data compression scheme. Inspired by the perspective of biological cognitive mechanism, we propose a cross-modal haptic compression scheme to improve the perception quality on VR devices in this paper. Since multimodal fusion is routinely employed to improve the ability of system in cognition, we assume that haptic codec is guided by visual semantics to strengthen data relevance in the coding process. We first design a multi-dimensional tactile feature fusion network (MTFFN), which extracts the multi-dimensional features in the material surface and maps them to infer a range of haptic coding parameters. Secondly, we provide second-order difference and linear interpolation to obtain an optimal criterion for the determination of codec parameters, which are inferred by the material surface characteristics. Finally, the simulation results reveal that our compression scheme can efficiently make a customized codec procedure for different materials, obtaining more than 17% improvement in terms of compression ratio with high immersive perceptual quality at the same time.