Revisiting Nonlocal Self-Similarity from Continuous Representation

Nonlocal self-similarity (NSS) is an important prior that has been successfully applied in multi-dimensional data processing tasks, e.g., image and video recovery. However, existing NSS-based methods are solely suitable for meshgrid data such as images and videos, but are not suitable for emerging off-meshgrid data, e.g., point cloud and weather data. In this work, we revisit the NSS from the continuous representation perspective and propose a novel Continuous Representation-based NonLocal method (termed as CRNL), which has two innovative features as compared with classical nonlocal methods. First, based on the continuous representation, our CRNL unifies the measure of self-similarity for on-meshgrid and off-meshgrid data and thus is naturally suitable for both of them. Second, the nonlocal continuous groups can be more compactly and efficiently represented by the coupled low-rank function factorization, which simultaneously exploits the similarity within each group and across different groups, while classical nonlocal methods neglect the similarity across groups. This elaborately designed coupled mechanism allows our method to enjoy favorable performance over conventional NSS methods in terms of both effectiveness and efficiency. Extensive multi-dimensional data processing experiments on-meshgrid (e.g., image inpainting and image denoising) and off-meshgrid (e.g., weather data prediction and point cloud recovery) validate the versatility, effectiveness, and efficiency of our CRNL as compared with state-of-the-art methods.

I. Introduction

With the rapid development of imaging and sensing technologies, numerous types of multi-dimensional data are readily available. Among them, signals with meshgrid structures can be modeled as arrays with multiple dimensions, e.g., a gray image can be represented by a matrix and a color image can be represented by a third-order tensor. Digging their intrinsic structures via hand-crafted techniques is an effective approach for many signal processing tasks. For example, real-world data usually has internal low-dimensional structures, and thus low-rank representation is a popular technique for data analysis and processing, e.g., inpainting [6], [7], denoising [8], [9], and compressed sensing [2], [10]. Besides, many data have intense local smoothness, which can be finely characterized by smooth regularization methods such as total variation [11], [12], [13] and smooth factorization [14], [15].