A 320 FPS Pixel-Level Pipelined Stereo Vision Accelerator with Regional Optimization and Multi-direction Hole Filling

Semi-global matching (SGM) is a canonical depth estimation algorithm widely used in the fields of autonomous driving, 3D reconstruction, and SLAM. However, high complexity withholds its application in high-speed and low-power depth extraction scenarios, notably in IoT and edge devices. To tackle these problems, we propose a region-optimized SGM algorithm, which can alleviate memory consumption bottleneck and strike a balance among power dissipation, processing speed, and resource consumption. First, we design a fully parallel initial matching costs calculation architecture, which ensures synchronization of the left and right pixel stream. Then, a two-layer parallel two-stage pipeline structure (TPTP) calculates the cost aggregation in two directions (0° and 135°) to mitigate the high computational complexity. Finally, we adopt a LUT-based cosine sub-pixel interpolation architecture and a multi-directional parallel hole filling architecture (MPHF) to improve accuracy further in the disparity refinement process. The experimental results show that the proposed pixel-level pipeline architecture achieves a processing speed of 320 frames per second (fps) at 98MHz on the Stratix-IV FPGA device.