Integrating human context and occlusion reasoning to improve handheld object tracking

Tracking an unknown number of various objects involving occlusion and multiple entry and exit points automatically is a challenging problem. Here we integrate spatial knowledge of human-object interactions into a high performing tracker to show that human context can further improve both detection and tracking. We use the DARPA Mind's Eye Action Recognition Dataset, which is comprised of street level scenes with humans interacting with handheld objects, to show this improvement. We find that human context can greatly reduce the number of false positive detections at the expense of increasing false negatives over a large test set (>230k frames). To minimize this, we add occlusion reasoning, where object detections are hallucinated when a human detection overlaps an object detection. These components together result in an average F
₁
improvement of 107% per object category and a 69% reduction in track latency.