Robust Portrait Image Matting and Depth-of-field Synthesis via Multiplane Images

High-quality portrait photography has become an essential function in our daily lives. However, due to the limited aperture and focal length of a smartphone camera, images captured by a smartphone cannot match the same level of bokeh effect by a digital single-lens reflex camera. A typical solution on a smartphone is to simulate out-of-focus effects from an all-in-focus image, where the key is robust depth estimation and portrait matting. To achieve this, we propose a multi-stage, multi-branch matting network to estimate a strand-level portrait alpha mask, which is then used to refine the coarse depth map obtained from the pre-trained model. Combining the input portrait image with the estimated depth map and alpha mask, we propose a learning-free optimization mechanism to construct a multi-plane image (MPI) representation for depth-of-field synthesis. The MPI consists of multiple layers of disk-blurred images with kernel size proportional to the absolute depth distance to the focus layer. Then a depth-aware blurring process is applied to enforce the bokeh effect. Besides, each MPI layer has an alpha channel controlling the visibility according to the corresponding depth. Finally, an image with bokeh is rendered by compositing all MPI layers. We conduct comprehensive experiments to evaluate our method, which demonstrates that our method can generate more accurate alpha masks and more realistic images with bokeh compared to prior work.