Optimizing Mobile-Friendly Viewport Prediction for Live 360-Degree Video Streaming

Viewport prediction is the crucial task for adaptive 360-degree video streaming, as the bitrate control algorithms usually require the knowledge of the user's viewing portions of the frames. Various methods are studied and adopted for viewport prediction from less accurate statistic tools to highly calibrated deep neural networks. Conventionally, it is difficult to implement sophisticated deep learning methods on mobile devices, which have limited computation capability. In this work, we propose an advanced learning-based viewport prediction approach and carefully design it to minimize transmission and computation overhead for mobile terminals. To improve viewport prediction accuracy, we utilize both spatial information through a saliency prediction model and temporal information through a modified LSTM model. Different computations introduced by the neural network models are distributed across the network to keep the computation light on mobile devices. To better adapt to the content dynamics in live streaming, we employ the model-agnostic meta-learning (MAML) method for video saliency prediction. The learned saliency prediction model with optimized initialization via offline meta-training can be fast fine-tuned online using a few samples. We further discuss how to integrate this mobile-friendly viewport prediction (MFVP) approach into a typical 360-degree video live streaming system by formulating and solving the bitrate adaptation problem. Extensive experiment results demonstrate that our approach achieves real-time prediction for live video streaming and surpasses existing methods in prediction accuracy on mobile terminals, which, together with our bitrate adaptation algorithm, significantly improves the streaming QoE from various aspects. Compared to baseline methods, MFVP achieves a 4.7–28.7% improvement in accuracy and demonstrates faster adaptability to dynamic content changes, enabling rapid fine-tuning and adjustment. When integrated into a streaming system and paired with our adaptive bitrate allocation algorithm, MFVP enhances overall video quality by 5.6–12.9% and reduces quality fluctuations by 33.3–50.9%.