On-Chip Depth and Image Sensing System With Offset Pixel Apertures

Depth sensing has been widely used in many applications over the past several decades. However, conventional depth sensing platforms are not suitable for mobile devices due to their large size and computation complexity. In this paper, we propose a compact system for depth and image sensing with offset pixel apertures (OPA). Specifically, we design an enhanced depth map processing (EDMP) and an OPA-dedicated image signal processing (OISP). The EDMP significantly improves the depth quality on excessively dark or light regions, greatly mitigates false depth aggregation at object boundaries, and increases depth precision with a very small overhead. Shot-noise-modeling and two-pass cost aggregation significantly reduce depth errors with reasonable hardware costs. Also, the OISP provides primary image processing while using white pixels to further improve color quality over conventional image sensors. Both EDMP and OISP are integrated on a single-chip. Our system shares the common hardware engine through time-multiplexing, taking advantage of the temporal margin caused by the sparsity of the OPA pixels. Compared to state-of-the-art OPA system, the proposed system reduces the depth error by 71.79%, increases the depth precision by 5.1 times, and reduces the angular error of color image by 54.08% on average.