Coupled Object Detection and Tracking from Static Cameras and Moving Vehicles

We present a novel approach for multi-object tracking which considers object detection and spacetime trajectory estimation as a coupled optimization problem. Our approach is formulated in an MDL hypothesis selection framework, which allows it to recover from mismatches and temporarily lost tracks. Building upon a multi-view/multi-category object detector, it localizes cars and pedestrians in the input images. The 2D object detections are converted to 3D observations, which are accumulated in a world coordinate frame. Trajectory analysis in a spacetime window yields physically plausible trajectory candidates. Tracking is achieved by performing model selection after every frame. At each time instant, our approach searches for the globally optimal set of spacetime trajectories which provides the best explanation for the current image and all evidence collected so far, while satisfying the constraints that no two objects may occupy the same physical space, nor explain the same image pixels at any time. Successful trajectory hypotheses are then fed back to guide object detection in future frames. The resulting approach can initialize automatically and track a large and varying number of objects from both static and moving cameras. We evaluate our approach on several challenging video sequences with both a surveillance-type scenario and a scenario where the input videos are taken from a moving vehicle.