A Hardware/Software Co-Design Approach for Real-Time Object Detection and Tracking on Embedded Devices

Many embedded applications require real-time visual analysis and sensing of the physical environment. Embedded vision is essential for a wide range of embedded applications such as video surveillance, robotics, and industrial manufacturing. While the algorithm design of vision applications have advanced significantly during the last decade, the embedded realization of many vision applications is at the early stages. The primary challenge is the real-time low-power execution of vision algorithms which demand high computation performance and high power consumption. This paper presents a novel hardware/software co-design approach for real-time execution of object detection and tracking on embedded devices. Our design targeted on Xilinx Zynq platform which combines the FPGAs reconfigurable fabric (for custom hardware implementation), and ARM CPU cores (for software implementation) in a single chip. The proposed approach consists of six major vision kernels including (1) Gaussian Smoothing, (2) Mixture of Gaussians Background Subtraction, (3) Morphology Filters all are mapped to the hardware, and Blob Detection, Histogram Checking, and Kalman Filter mapped to ARM cores (software execution). All six vision kernels execute concurrently over streaming pixels in a producer/consumer fashion. Our results, based on implementation and full integration on Xilinx Zynq platform, presents a real-time performance on 780P pixel resolution at 60 frames per second with less than 2 Watt power consumption.