A 0.8V Intelligent Vision Sensor with Tiny Convolutional Neural Network and Programmable Weights Using Mixed-Mode Processing-in-Sensor Technique for Image Classification

Vision systems with artificial intelligence (AI) for applications requiring image classification are in growing demand. However, the imager plus dedicated AI accelerator solution [1] suffers from the burdens of power and latency caused by the raw image data traffic between the imager and the companion signal processor with a neural network accelerator, making it unsuitable for the real-time inference in low-power edge devices. Recently, imagers with near- or in-sensor processing capability have been developed [2]–[6] to improve the system efficiency for specific applications. In [2]–[4], the near-sensor Haar-like filtering operations are implemented in imagers to realize face detection (FD). However, unlike using convolutional neural networks (CNNs) with programmable weights for different tasks, the implemented features of such prior works are limited and not configurable. In [5], a convolutional CMOS image sensor (CIS) with near-sensor analog multiply-accumulate (MAC) operations was reported for assisting with the 1st-layer computations of a CNN. However, the convolutional CIS is inadequate for some tasks, due limits on the numbers of layers/kernels, and needs a companion digital accelerator for the required operations (Rectified Linear Unit: ReLU, Maximum-Pooling: MP, Fully-Connected layer: FC, etc.) of a complete CNN model. In [6], an analog convolutional CIS is reported with a 5-layer network for CNN implementation. However, the analog MAC operations using charge sharing with a capacitor array leads to gain loss, low weight resolution, and limited accuracy. Moreover, the ReLU+MP operation using a static winner-take-all circuit is power hungry. To address these issues, we present an intelligent vision sensor (IVS) with an embedded tiny CNN model and programmable weights to achieve configurable feature extraction and on-chip image classification using a mixed-mode processing-in-sensor (PIS) technique.