A Visual-Cortex-Mimetic Tiny Neuromorphic Vision Processor Based on Reconfigurable Cortical Neuron Unit

Edge visual systems demand high energy-efficiency vision processors like neuromorphic hardware leveraging spike-based computations. But their disability of directly interacting with non-spike information in the real world requests additional components to execute image pre-processing, spike encoding and decoding, severely increasing overall system cost, energy and latency. To overcome such drawback, this brief proposes a tiny neuromorphic vision processor which emulates functional regions along the ventral pathway in the visual cortex. It performs image pre-processing, spike encoding, spike-based feature extraction and classification, spike decoding as well as decision making on a single chip. To reduce hardware resources, our processor builds on a reconfigurable cortical neuron (RCN) unit, which runs different neuron models for different visual cortex regions in a time-multiplexing fashion. It also embeds biological learning circuits to better adapt the processor to dynamic edge scenarios. Our neuromorphic processor was prototyped on a very-low-cost Xilinx Zynq-7010 device. On the MNIST dataset, it exhibited a real-time inference speed of 696 frame/s covering image pre-processing to final decision and a high on-chip learning accuracy of 97.12%, while only delivering a power consumption as low as 118 mW.