Video Inpainting by Fusing Optical Flow with Spatial-Temporal Transformer

In recent years, optical flow and transformer have been applied to video inpainting successfully. However, in the previous works, the networks for extracting and completing optical flow need to be trained separately, which is a cumbersome process. Meanwhile, the transformer architecture suffers from high computational complexity and imprecise attention retrieval. Therefore, a video inpaiting model by fusing optical flows with a spatial-temporal transformer is proposed in this paper. In this model, a novel cross-stacked spatial-temporal transformer is designed, utilizing distant frames as reference frames within the temporal transformer. Simultaneously, motion information from optical flows is integrated into the spatial transformer to refine the attention retrieval. Within these transformer blocks, a window focal attention mechanism is introduced to achieve a combination of coarse-grained and fine-grained attention. This model maintains inpainting quality while lowering computational complexity effectively. Additionally, by training the flow completion network and other modules jointly, an end-to-end inpainting process is achieved. The effectiveness and superiority of the proposed model are confirmed by both quantitative and qualitative evaluation results.