3DSEAVNet: 3D-Squeeze-and-Excitation Networks for Audio-Visual Saliency Prediction

Video saliency prediction is an important task in the field of computer vision. Most of the existing video saliency prediction methods only focus on image information, and the audio information is often ignored. This leads to an incomplete perception mode, which makes it difficult to achieve optimal performance. SENet is an excellent attention mechanism-based network. It significantly enhances the performance of 2D convolutional networks. However, whether the 3D convolutional network can be applied to this attention mechanism network remains to be studied. In order to solve the above problems, we propose a saliency prediction network for audio-visual fusion to extract and predict various information in videos. At the same time, we improve the traditional SENet to make it applicable in 3D convolutional neural networks and discuss its role. Compared with the state-of-the-art methods, our model has strong competitiveness in multiple data sets.

I. Introduction

Saliency prediction has received a lot of attention recently in the field of computer vision. It is well known that humans have the ability to rapidly analyze and interpret the complex environment around them and to focus their attention on a certain area. In order to make machines have this intelligent behavior as well, saliency prediction techniques are born on demand. The main purpose of video saliency prediction is to model human attention and then predict the areas where human attention is likely to be focused while watching a video. Previous studies have shown that this technology can be applied to many scenarios, such as visual tracking [1]–[2], video compression [5], video subtitles [3]–[4], etc.