Video Event Detection via Multi-modality Deep Learning

Detecting complex video events based on audio and visual modalities is still a largely unresolved issue. While the conventional video representation methods extract each modality ineffectively, we propose a regularized multi-modality deep learning for video event detection. We first build an auto-encoder based on unconstrained minimization and adopt the conjugate gradient method with linear search for optimization. The learned auto-encoder can capture the relationship between the audio and visual modality corresponding to the same video event at each layer of the network. To make the network robust to local variance, we adopt the commonly used local contrast normalization and spatial maximum pooling to each modality for video representation. Compared with traditional methods using manually designed features, our method is more efficient. Experimental results on publicly available video event detection datasets demonstrate that the proposed method consistently outperforms the state-of-the-art video representation methods.