Sound Quality Improvement of Extracted Sound from Video with Rolling-shuttered Camera

Recent researches have proposed to photograph surface vibration of object with a rolling-shuttered camera to extract sound from video. These methods measure the surface of an object that vibrates according to target sound, which are more robust to noise than air conduction microphones. However, the rolling-shuttered camera has the periods in which the camera cannot photograph and they cause gaps in the extracted sound. Also, noise is mixed in the extracted sound during the sound extraction process. As a result of them, the sound quality of the extracted sound deteriorates. Therefore, we propose sound quality improvement of extracted sound by interpolation and noise suppression technique. In the interpolation, sound is restored using the fundamental frequency of the backward signal of the gap, and in the noise suppression, sound quality is improved by applying a widely used noise suppression method. Finally, experimental results showed that the proposed method increased the PESQ score by 0.3 on average.