A 10 000-Inference/s Bio-Inspired Spiking Vision Chip Based on an End-to-End SNN Embedding Image Signal Enhancement

Edge machine vision requires a vision chip to perform imaging and visual recognition with high speed, high accuracy, and compact size. Current vision chips adopt a two-stage workflow, which uses computer vision processing and a deep neural network (DNN) to perform signal enhancement and recognition, respectively. However, the two stages involve different operations and are separately optimized, which causes degraded recognition accuracy, low inference rate, and huge hardware overhead. To address it, we propose a vision chip that integrates a single-photon avalanche diode (SPAD) image sensor and a reconfigurable spiking neural network (SNN) processor. Inspired by the human visual system that employs a full SNN for signal enhancement and recognition, the vision chip acquires spike-based images and performs an end-to-end SNN embedding image signal enhancement (ISE) for object recognition. The ISE and recognition are trained jointly in a unified deep learning framework to increase recognition accuracy and inference speed. The vision chip only integrates an SNN processor to perform ISE and recognition by being reconfigured into the low-precision multiply-accumulation (MAC) mode or accumulation (ACC) mode, respectively, thereby avoiding a dedicated image signal processor (ISP). Based on a 55-nm CMOS technology, the fabricated vision chip integrates $128{\times }128$ pixels and can implement 4096 ACC/MAC components. The chip was measured to exhibit: 1) a 99.2% accuracy with 10000 inferences/s at 10 lx and 95.3% at 1.43 mlx on the MNIST dataset and 2) a 94.32% accuracy with color mosaic images on the ETH-80 dataset. Our chip maintains a relatively high recognition performance even under an illumination one order of magnitude lower than in prior vision chips.