Real time moving vehicle detection and reconstruction for improving classification

Vehicle images captured by traffic and surveillance video cameras in various conditions usually exhibit several unexpected variations that worsen vehicle classification. These factors include occlusions, motion blur, and changes in perspective views. Complete and normalized views of vehicle images, if being able to reconstructed from the unsatisfactory data, will facilitate more accurate data labeling, feature extraction and multi-class vehicle classification. We propose a multimodal temporal panorama (MTP) approach to accurately extracting and reconstructing moving vehicles in real-time using a remote multimodal (audio/video) monitoring system. The MTP representation consists of: 1) a panoramic view image (PVI) for detecting vehicles using the concept of 1D vertical detection line; 2) an epipolar plane image (EPI), generated from 1D epipolar lines along the vehicles' moving paths, to characterize their speeds and directions; and 3) corresponding audio signals collected at the vehicle detection point to reduce false target detection in the PVI. Using the MTP approach, reconstructed vehicles all have the same side views, with less or no occlusions and motion blur. Using SVM classifiers for multiclass problems indicates that the classification accuracy using reconstruction results improves about 10% over that using corresponding vehicle images from original video for a dataset of about 140 vehicles. Our ultimate goal is to use the audio-visual vehicle data for multimodal vehicle classification and anomaly detection.