Video-Rate Stereo Matching Using Markov Random Field TRW-S Inference on a Hybrid CPU+FPGA Computing Platform

We demonstrate a video-rate stereo matching system implemented on a hybrid CPU+field-programmable gate array (FPGA) platform (Convey HC-1). Stereo matching is a fundamental problem of computer vision, and emerging applications, such as 3-D gesture recognition and automotive navigation, demand fast and high-quality stereo matching. Markov random field (MRF)-based approaches are widely used, but conventional software solvers are slow. Belief propagation (BP) solvers, which use patterns of local message passing on MRFs, have been studied in hardware, but their performance is unreliable. We show how a superior method, sequential tree-reweighted message passing (TRW-S), can be rendered in hardware. TRW-S has reliable convergence, guaranteed by its so-called sequential computation. Analysis reveals many opportunities for TRW-S hardware acceleration. Starting from the core architecture for streaming TRW-S, we describe the end-to-end system engineering for full video stereo matching. We partition the stereo matching procedure across the CPU and the FPGAs and apply frame level optimizations, such as message reuse based on scene change detection, frame level parallelization, and function level pipelining. The experimental results show that our system achieves a speed of 22.8 frames/s for a challenging QVGA video stereo matching task. We noticed that our system is significantly faster than several recent GPU/ASIC implementations of similar stereo inference methods based on BP.