Towards multimodal saliency detection: An enhancement of audio-visual correlation estimation

This work is focused on the task of multimodal saliency detection. Very few works have been developed in the field, and there are no well-established baselines or benchmarks comparable to those existing in the field of visual saliency detection. In this paper, we set out to improve an existing model by enhancing the performance of its key module: the audio-visual correlation estimation based on the computation of the correlation of its acoustic inconsistency with visual motion estimation. We propose two main modifications: first, the introduction of auditory saliency algorithms to compute the acoustic inconsistency and second, the improvement of the motion estimation procedure. To verify our hyphothesis and analyze the performance of our contribution, we have developed experiments considering a database of 30 videos, that includes eye-tracker fixations of several subjects. Results show that using acoustic saliency improves the performance of the system compared with a conventional visual saliency model with color, orientation, intensity and motion estimation. It also outperforms the baseline audio-visual system, that uses acoustic inconsistency as an auditory cue. On the other hand, using optical flow for motion estimation also increases significantly the performance of the system, regardless of the tested acoustic feature.