Work-in-Progress: Temporal RegionDrop – Frame Difference Sparsity for Efficient Video Inference

Convolutional Neural Networks (CNNs) require costly hardware for real-time processing due to massive multiplications. In video-based inference, many pixels remain unchanged between frames. By skipping these operations and reusing previous results, CNNs can be accelerated without accuracy loss. However, efficient implementation of sparse operations is challenging due to the need for change detection and sparse operations in each layer. To address this, we propose Temporal RegionDrop, which performs change detection once per frame and uses a common mask to reduce conversion costs between sparse and dense formats. This method is applicable to all CNNs without retraining. We evaluate our RegionDrop on AGX Orin GPU with ChangeDetectionNet, MOT15, and JHMDB datasets. RegionDrop achieves a 2.0× –6.5× speedup with minimal accuracy loss.