SensorFlow: Sensor and Image Fused Video Stabilization

We present SensorFlow, a novel image and sensor fusion framework for robust, high-quality video stabilization. We start with sensor-based pre-stabilization that smooths out large-scale camera motion. A new angular velocity domain optimization has been introduced to achieve frame rate in-variance. We then feed the stabilized optical flows into an occlusion-aware 3D CNN that infers dense warp fields to remove residual translation and jitter. To further avoid dis-tortion, we propose a novel masking scheme to determine the disoccluded and dynamic regions in optical flow and in-paint them with spatially smooth flow vectors. Our method is appealing as it shares both the dense warping field's flex-ibility to correct complex motions and the robustness of sen-sor data for arbitrarily challenging scenes. We have vali-dated its effectiveness and demonstrated our solution out-performs state-of-the-art alternatives via extensive ablation studies and quantitative comparisons.