Enhancing Aerial Cinematography: Robust Visual-Inertial Tracking for Agile Subjects with Novel Data Integration

Aerial cinematography via autonomous robots faces significant challenges, such as maintaining subject tracking under harsh conditions. This work addresses continuous visual tracking of agile subjects via aerial robots, even when the subject is occluded or out of the camera's field of view. Our approach integrates inertial and visual data for enhanced tracking robustness. We perform stochastic extrapolation of the subject's path, creating probability images that highlight potential future positions using color coding on a map of the filming scene. These images, derived from a 3D reconstruction of the scene, guide a Siamese-network-based tracker to locate the subject when tracking is lost. We conducted extensive real-world flight tests to collect a novel dataset, encompassing visual and inertial data from both the aerial robot and the subject. Our visual-inertial-based tracker correction method, evaluated on this dataset, demonstrated significant improvements over purely visual deep trackers and existing research, showcasing its potential for advanced applications in aerial cinematography. Our software and dataset are available online¹¹https://github.com/hamidrezafahimi/pyTrackers.