Temporal Frame Filtering with Near-Pixel Compute for Autonomous Driving

As the computer vision techniques behind the advances in autonomous driving become more complex, the on-vehicle processors experience increased burden due to the need to compute on the received images in real-time. Near-pixel compute is an emerging paradigm that brings part of the critical compute tasks very close to the image sensors, and the transmitted data is reduced in dimensionality or partially processed to alleviate the backend compute stress. In this work, we present a near-pixel compute architecture targeting a temporal frame filtering network which drops redundant image frames for the autonomous driving BDD100K dataset. Several circuit optimizations to the compute units are introduced, including using NOR gate array as data buffers, front-end demosaicing, sparsity-aware adder tree, and indirect calculation of image frame difference. The proposed design is synthesized at 40 nm, a node compatible with CMOS image sensor (CIS) technology, which completes the filtering algorithm in 2.2 ms with an energy efficiency of 15.7 TOPS/W and compute density of 380.1 GOPS/mm².