Neuromorphic Vision Restoration Network for Advanced Driver Assistance System

Adverse weather conditions severely impact the environmental perception capabilities of the Advanced Driver Assistance System (ADAS). However, it is difficult to deploy these high-performance Transformers for image restoration on resources constrained ADAS platform. Recently, the rapid development of neuromorphic computing systems (NSC), based on fast in-memory computing, provides an ideal and feasible solution to it. Combining Transformers and NSC faces challenges due to the high complexity and outliers of Softmax attention. Moreover, existing image restoration networks are mostly designed for a single task and lack generalization. To address these issues, we propose a neuromorphic-computing-friendly Linear Sparse Transformer (LSFormer) with linear computational complexity. The LSFormer achieves advanced image restoration performance compared to Softmax attention Transformers. Meanwhile, we propose the implementation scheme based on neuromorphic computing systems, reducing the deployment complexity at the edge. We evaluate LSFormer on multiple image restoration tasks including low-light image enhancement, image deraining, and dehazing. The results demonstrates that the performance of LSFormer outperforms state-of-the-art (SOTA) networks. Furthermore, the comparison between existing hardware implementations of Softmax attention and our proposed scheme using various metrics validates the feasibility and superiority of our proposed scheme.