A Real-Time Event Vision Sensor based Object Detection and Tracking System for Edge Applications

Recently, visual object detection and tracking (ODT) has become ubiquitous in intelligent edge platforms. RGB camera-based ODT systems using deep learning models [1] - [2] achieve high accuracy at the cost of significant power consumption. In contrast, event vision sensors (EVS) dramatically reduce the data volume, as events are generated only for moving objects. However, accurate object classification in event-based object tracking systems [5] – [7] is challenging due to the lack of detailed texture information. This paper proposes an energy-efficient ODT system that uses both the event stream and grayscale image from the EVS, featuring:1) A memory-efficient adaptive frame builder for event binary image (EBI) reconstruction without motion blur; 2) A memory-efficient region proposal unit for object localization with low latency; 3) A fully pipelined CNN engine with input sparsity enhancement and customized instruction set architecture (ISA).