IENet: Internal and External Patch Matching ConvNet for Web Image Guided Denoising

From the non-local self-similarity (NSS)-based image denoising to the convolutional-network (ConvNet)-based image denoising, the denoising performance has been greatly improved. However, it is still not clear how to utilize similar web images to guide image denoising using ConvNet. This paper proposes a novel ConvNet for image denoising to explore both internal (NSS) and external correlations when external similar images are available. Since external similar images may be taken with different viewpoints, focal lengths, and may contain different objects, it is difficult to directly explore external correlations at image level using ConvNet. Therefore, we propose an internal and external patch matching ConvNet (IENet), whose inputs are similar patch cubes extracted from the noisy input and its external similar images. We design three different network structures, namely early-fusion, middle-fusion, and late-fusion of the internal and external cubes to fully combine the strengths of internal and external correlations. The experimental results demonstrate that the proposed method achieves the best denoising results compared with the seven state-of-the-art denoising methods. In specific, the proposed method outperforms the state-of-the-art web image guided denoising method by more than 1 dB on average, which further demonstrates the superiority of the proposed IENet-based filtering over the hand-crafted filtering methods.