SwinSpikeFormer: Learning Comprehensive Spatial-Temporal Representation to Reconstruct Dynamic Scenes from Spike Streams

The spike camera, with its high temporal resolution, low latency, and high dynamic range, addresses high-speed imaging challenges such as motion blur. It captures photons at each pixel independently, creating binary spike streams rich in temporal information. However, these binary spike streams present a challenge for their reconstruction into images, which is necessary for performing subsequent downstream tasks. Current algorithms, both traditional and deep learning-based, fail in utilizing this rich temporal information and restoring the details of the reconstructed image. To tackle this challenge, we propose Swin Spikeformer (SwinSF), a novel model for dynamic scene reconstruction from spike streams. SwinSF is composed of three main modules: Spike Feature Extraction, Spatial-Temporal Feature Extraction, and Final Reconstruction Module. The Spike Feature Extraction Module uses convolution layers to directly extract features from the spike streams. The Spatial-Temporal Feature Extraction Module includes several Residual Swin Spikeformer Blocks (RSSB), which integrate shifted window self-attention and proposed Temporal Spike Attention (TSA) to effectively extract both spatial and temporal information from intra-frames and inter-frames, leading to a more robust and accurate reconstruction of spike streams. The Final Reconstruction Module fuses both spike and spatial-temporal features to produce the final image reconstruction. Furthermore, we have built a new synthesized dataset for spike image reconstruction, featuring high resolution and enhanced temporal fidelity through the use of frame interpolation techniques, which ensures its relevance and applicability to realistic spike camera imaging scenarios. Experimental results demonstrate that our proposed network, SwinSF, sets a new benchmark, achieving state-of-the-art performance across a series of datasets, including both real-world and synthesized data across various resolutions. Our codes and proposed dataset can be found at https://github.com/bupt-ai-cz/SwinSF.