Custom processor design for efficient, yet flexible Lucas-Kanade optical flow

State-of-the-art solutions to optical flow fail to jointly offer high density flow estimation, low power consumption and real time operation, rendering them unsuitable for embedded applications. Joint hardware-software scalability at run-time is crucial to achieve these conflicting requirements in one device. This paper therefore presents a scalable Lucas-Kanade optical flow algorithm, together with a flexible power-optimized processor architecture. The C-programmable processor exploits algorithmic scalability through innovations in its memory structure, memory interface, and datapath optimized for efficient convolutions. Jointly, the scalable flow algorithm and optimized computer vision hardware platform enable applications to on-the-fly trade-off throughput and power consumption in function of flow density and accuracy. The processor chip is synthesized in 40nm CMOS technology and verified on FPGA. The architecture is capable of scaling the frame rate at run-time and processes 16fps of dense optical flow at 640×480 resolution with 15.06° average angular error, while only consuming 24mW.