Reliable intruder detection using combined modalities of intensity, thermal infrared and stereo depth

The task of large-area visual monitoring for the protection of critical and public infrastructures calls for reliable automated visual surveillance systems. Reliability in this context implies that a high detection accuracy of critical events shall be maintained independent from observation conditions, appearance and pose variations of observed objects (persons, cars), while accomplishing a low-rate of false alarms. In this paper we propose a real-time intruder detection system based on a combination of multiple complementary sensor modalities. The proposed system employs a trinocular stereo setup consisting of intensity and thermal infrared cameras capable to cover an observation area of 60m deep × 20m wide where modalities of intensity, stereo depth and thermal infrared are measured and combined to detect, track and classify objects entering the observed area. The individual modalities are combined using an estimated 3d ground plane as a common reference, yielding a probabilistic occupancy map for object candidates. A fast non-parametric clustering technique well coping with noise and multiple nearby objects is used to delineate objects, taking scale information, given the ground plane, into account. The proposed system is validated on two challenging video sequences. Promising intruder detection results are presented in terms detection and false alarm rates.