ISCom: Interest-Aware Semantic Communication Scheme for Point Cloud Video Streaming on Metaverse XR Devices

In the metaverse era, point cloud video (PCV) streaming on mobile XR devices is pivotal. While most current methods focus on PCV compression from traditional 3-DoF video services, emerging AI techniques extract vital semantic information, producing content resembling the original. However, these are early-stage and computationally intensive. To enhance the inference efficacy of AI-based approaches, accommodate dynamic environments, and facilitate applicability to metaverse XR devices, we present ISCom, an interest-aware semantic communication scheme for lightweight PCV streaming. ISCom is featured with a region-of-interest (ROI) selection module, a lightweight encoder-decoder training module, and a learning-based scheduler to achieve real-time PCV decoding and rendering on resource-constrained devices. ISCom’s dual-stage ROI selection provides significantly reduces data volume according to real-time interest. The lightweight PCV encoder-decoder training is tailored to resource-constrained devices and adapts to the heterogeneous computing capabilities of devices. Furthermore, We provide a deep reinforcement learning (DRL)-based scheduler to select optimal encoder-decoder model for various devices adaptivelly, considering the dynamic network environments and device computing capabilities. Our extensive experiments demonstrate that ISCom outperforms baselines on mobile devices, achieving a minimum rendering frame rate improvement of 10 FPS and up to 22 FPS. Furthermore, our method significantly reduces memory usage by 41.7% compared to the state-of-the-art AITransfer method. These results highlight the effectiveness of ISCom in enabling lightweight PCV streaming and its potential to improve immersive experiences for emerging metaverse application.

I. Introduction

Point cloud videos (PCVs), characterized by 3D unordered points with RGB color or mesh formats, facilitate an interactive and immersive experience by providing 6-degrees-of-freedom (6-DoF) movement in the metaverse [1]. Real-time streaming of volumetric PCVs imposes new challenges on the existing network infrastructure and codecs, such as exceptionally high bandwidth and substantial computational requirements for codecs. Firstly, PCVs exhibit characteristics of enormous data volumes and require ultra-high bandwidth for streaming. Taking a popular Kinect depth camera for capturing as example, it generates 2.06 gigabits (Gb) of raw data per second at a frame rate of 30 frames per second (FPS) [2]. As more cameras are incorporated to achieve finer PCV content, the demand for higher bandwidth escalates [3]. Secondly, existing codec solutions based on MPEG extensions remain focused on efficient compression and quality assurance of offline PCVs, with a lack of research into real-time PCVs [4]. This is particularly evident in the absence of studies on adaptive PCV transmission when dealing with dynamic network environments and computational heterogeneous mobile devices. Consequently, the immense computational demands of codecs and dynamic streamings hinder the delivery of a 6-DoF experience on mobile metaverse devices, such as augmented reality (AR) and virtual reality (VR) headsets.