Specular Detection and Rendering for Immersive Multimedia

Immersive multimedia has received a lot of attention because of its huge impact on user experience. To realize high immersion in virtual environments, many virtual views should be generated at arbitrary viewpoints with advanced display devices. However, specular regions, which affect user experience, have not been fully investigated in an immersive multimedia field. In this article, we propose specular highlight detection and rendering methods to improve immersion. For specular detection, a high-performance variational attention U-network (VAUnet), which combines a variational autoencoder and a spatial attention mechanism, is proposed with a hybrid loss function. The specular regions detected from VAUnet are compressed with an immersive video coding standard (MPEG-I), and then the rendering is performed by considering the decompressed specular regions. Extensive experiments demonstrate that the proposed method improves specular detection performance and subjective rendering quality.