A Configurable and Real-Time Multi-Frequency 3D Image Signal Processor for Indirect Time-of-Flight Sensors

3D time-of-flight (ToF) cameras have recently attracted widespread attention because of their potential in wide applications. However, the lack of versatile 3D image signal processor (ISP) is still an obstacle that hinders their popularity. The main challenges lie in the management of high-throughput raw data and the complexity of 3D image processing in hardware. To address these challenges, this paper presents a top-down 3D ISP with high real-time performance and high energy efficiency for indirect ToF (IToF) imaging. First, we propose configurable cache strategies to improve the efficiency of non-streaming memory accesses and support different IToF sensors. Then, to resolve the phase ambiguity problem commonly existing in IToF sensors, we develop a hardware-optimized cascaded multi-frequency fusion method, which not only supports up to six modulation frequencies but also facilitates the pixel-wise detection of multipath interference. Moreover, we propose a scalable calculation engine for depth calculation and calibration, 3D image enhancement, and post-processing. The pipeline is prototyped on a Xilinx ZCU102 FPGA board. Results show that the 3D ISP has negligible precision loss and low latency, achieving
$53.35\times $
energy efficiency improvement compared with the implementation on Intel CPU and
$55.45\times $
improvement over the ARM CPU implementation. When running at 75 MHz, the 3D ISP can support
$1280\times 960$
depth, intensity, and point cloud output at a frame rate of 30 f/s.