Feedback scheme for thermal-visible video registration, sensor fusion, and people tracking

In this work, we propose a feedback scheme for simultaneous thermal-visible video registration, sensor fusion, and tracking for online video surveillance applications. The video registration is based on a RANSAC trajectory-to-trajectory matching that estimates an affine transformation matrix that maximizes the corresponding trajectory points and overlapping of foreground thermal and visible pixels. Sensor fusion uses the aligned images to compute sum-rule blobs for thermal and visible images and constructs the thermal-visible blobs. Finally, the multiple object tracking gets blobs constructed in sensor fusion as the input and outputs the trajectories of moving humans in the scene. We tested our method on long-term indoor and outdoor video sequences and demonstrate the effectiveness of our feedback design in obtaining better quality for both image registration and tracking.