An FPGA-based Ultra-High Performance and Scalable Optical Flow Hardware Accelerator for Autonomous Driving

Optical flow plays an extremely important role in the field of computer vision and extremely high real-time performance is required especially in autonomous driving. Traditional optical flow methods generally improve the accuracy of optical flow through the image pyramid technique. However, the introduction of the image pyramid increases the computational complexity. Additionally, the data relationships between pyramid layers result in strong data dependencies, making it difficult to accelerate using parallel processing, and it is challenging to meet real-time requirements in practical scenarios. To address this issue, in this paper, we propose an FPGA-based ultra-high performance and scalable optical flow hardware accelerator with several techniques, including an adaptive optical flow computation technique based on dynamic direction prediction to reduce computation without accuracy degradation, a highly scalable computing architecture with configurable numbers of PEs to improve the flexibility and hardware utilization under different hardware resource constraints, and a reconfigurable pyramid-layer pipeline technique to improve performance and reduce memory size. The proposed hardware accelerator was implemented and evaluated on a Xilinx FPGA ZCU104 achieving ultra-high performance (405 FPS) while maintaining high accuracy (AEE 0.64).