Binauralization Robust To Camera Rotation Using 360° Videos

We propose a novel binauralization method that is robust to camera rotation. Since binaural audio can bring a 3D sensation to the listener, it can enhance the immersive experience of the video. Researchers have been explored binaural audio generation from monaural audio to deepen the experience of already captured videos without special recording devices. However, this binauralization on real-world videos can be a challenging task due to camera rotation. Camera rotation makes it difficult to predict the exact sound source position due to the motion of the sound sources and the background in videos. To tackle this problem, we propose a training data generation pipeline using 360° videos for binauralization. We generate monocular videos and binaural audio with camera rotation from 360° videos for the training of binauralization. Additionally, we newly construct a binauralization framework that conducts multi-task learning with camera localization. The camera localization predicts the camera rotation and helps the binauralization. Experimental results show that our method can achieve the binauralization on videos with camera rotation.

1. INTRODUCTION

The experience of viewing videos has evolved through the advancement of video recording and viewing devices: from black-and-white videos to color videos, from television to smartphones. Nowadays, egocentric videos and 360° videos are being collected [1],[2] and studied for use in VR/AR technology [3], robotic navigation [4], skill acquisition [5], etc., and are expected to become widespread in the future. Egocentric videos are recorded from a human viewpoint by following the motion of the human body and head, and 360° videos capture a view in every direction using a omnidirectional camera and allow users to control the viewing direction. These videos are close to human vision since humans have a bounded field of view and rotate their heads to obtain information about their surroundings [6].