Multi-Memory Streams: A Paradigm for Online Video Super-Resolution in Complex Exposure Scenes

Existing online video super-resolution methods utilize implicit memories of previous frames to provide reference information, which have a single memory stream path and are highly dependent on the continuous memory stream. However, video capture in real-world scenes is typically affected by abnormal exposures resulting in sudden changes of lightness thus interrupting the memory stream, while long-term memories suffer from memory vanishing problems during transmission. To address this problem, we propose a novel multi-memory streams based online video super-resolution paradigm that adaptively corrects for abnormal exposures and creates multi-memory streams to accurately converge long-term memories. Specifically, we first propose an exposure detection-correction module, which utilizes optical flow overfitting property and temporal lightness information to detect and correct abnormal exposures to avoid interruption of memory streams. In addition, we propose a dynamic-static decoupled alignment strategy, which can adaptively select the alignment method based on pixel displacement, thus accurately aggregating past long-term memories to create multiple memory streams. Further, we propose an adaptive memory fusion module to mine complementary information between multiple memory streams to solve the memory vanishing problem. Extensive experimental results show that our method outperforms existing video super-resolution methods on complex exposure datasets. We also conduct detailed ablation experiments to analyze and validate our contributions. The implementation code is available at https://github.com/GZ-T/MMVSR.