Pyramid Matting: A Resource-Adaptive Multi-Scale Pixel Pair Optimization Framework for Image Matting

Image matting is an important problem in computer vision with signiﬁcant theoretical interest and diverse practical applications, including image/video editing, target tracking, and object recognition. Pixel-pair-optimization-based image matting approaches have been shown very successful in estimating the opacity of the foreground by searching for the best pair of foreground and background pixels for each unknown pixel. However, extant approaches encounter difﬁculties in adapting to the changes of available computing resources, which limits the application of image matting. This drawback has motivated the present study, as a part of which a multi-scale evolutionary pixel pair optimization framework named pyramid matting framework (PMF) was developed. In this framework, the large-scale pixel pair optimization problem is transformed to multiple pixel pair optimization problems of different scales using image pyramid. The resulting problems are solved level by level, starting from the problem at the small scale. Pixel pair heuristic information obtained from solving low-scale problems are iteratively propagated to the spatially-related pixel pairs in the larger-scale problem. PMF can adapt to changes in available computing resources due to its capability of transforming a small-scale problem solution to the large-scale problem solution through the heuristic information propagation. Experimental results show that the PMF-based image matting approach not only provides high-quality alpha mattes with sufﬁcient computing resources, but also works well when computing resources are scarce.


I. INTRODUCTION
Natural image matting aims to accurately extract foreground objects from a color image with a natural background. The The associate editor coordinating the review of this manuscript and approving it for publication was Shiqi Wang. extraction is implemented through the estimation of alpha matte (foreground opacity) based on user-defined information (typically provided in the form of trimap). The natural image matting model can be mathematically described as: where the observed color (I p ) of a pixel p is denoted as a convex combination of a foreground color (F p ) and a background color (B p ); α p is the weight of the color composition (i.e. the opacity of the foreground) at pixel p. Hence, α p ranges from 0 to 1, whereby 1 denotes definite foreground pixels and 0 represents definite background pixels. A trimap separates an image into three non-overlapping regions, denoted as: unknown, known foreground and known background regions. The aim of natural image matting is to determine the alpha values of pixels in unknown regions, while those in known foreground and known background regions are denoted as 1 and 0, respectively. Natural image matting has been applied not only to mid-level vision tasks such as image fusion [1], automatic foreground extraction [2], [3], semantic segmentation [4], medical image processing [5] and target tracking [6], but also to high-level vision tasks, such as pedestrian classification [7], object recognition [2], virtual reality [8], and street view augmented reality [9]. As the range of image matting applications broadened, the image matting approach is required to provide satisfactory alpha mattes within a given amount of computing resources. However, the amount of available computing resources changes with the time sensitivity of the image matting task in the applications mentioned above. For example, an image matting approach may be required to provide an alpha matte within seconds in a time-sensitive task, while it can be run for hours in a time-insensitive task.
The pixel-pair-optimization-based approach is one of the competitive image matting approaches that have distinct advantages in parallelization [10] and in handling a mislabeled trimap [7] or spatially disconnected foreground [11]- [13]. In this approach, natural image matting problem is modeled as a pixel pair optimization (PPO) problem that can be written as: where g p (x p ) is a pixel pair evaluation function for an unknown pixel p; U, F and B are pixel sets in unknown, known foreground and known background regions, respectively; x p is the pixel pair decision vector for pixel p; and x p denote a pixel in known foreground and background regions, respectively. Once the foregroud and background colors are given by solving the PPO problem, the alpha valuê α p for pixel p can be estimated by applying the following expression:α where || * || 2 denotes Euclidean norm of vector *.
The PPO problem remains a significant research topic because of three major difficulties. (1) the pixel pair objective function is non-convex and has lots of local optima, as shown in Fig. 1; (2) substantial optimization problems need to be FIGURE 1. Pixel pair fitness regarding a small pixel pair decision subspace of the PPO problem pertain to an unknown pixel in an image named GT25. The fitness is obtained by a pixel pair evaluation function proposed in [14].
solved with limited computing resources (for example, hundreds of thousands of optimization problems can be involved in a 0.5 megapixel image); and (3) the PPO problem search space is extensive. As all combinations of the pixels in known foreground and known background regions constitute the set of possible pixel pairs, the number of possible pixel pairs for one unknown pixel in a 0.5 megapixel image could reach 10 8 .
Considering the change of available computing resources in image matting applications, the PPO problem is required to be solved approximately with different amounts of available computing resources. The amount of available computing resources can be quantitatively described by the number of available pixel pair evaluations per unknown pixel (PPE/UP) [15]. The PPO-based approaches described in pertinent literature can be categorized into: sampling-based and evolutionary-optimization-based approaches.
Sampling-based approaches reduce decision space with sampling to approximately solve the PPO problem. In most recent studies, alpha mattes are estimated by pixel sampling and pixel pair optimization [14]. Two small sets of pixels are collected from known foreground and known background regions separately to reduce the PPO problem search space. Subsequently, the best pixel pair for each unknown pixel is chosen from the sample sets by optimizing a pixel pair evaluation function.
Researchers primarily focused on designing pixel sampling strategies based on spatial relationships in earlier studies on sampling-based approaches. Wang and Cohen [16] assumed that the known foreground or background pixels in close proximity to the unknown pixel may constitute the best pixel pair. In the approach proposed by Gastal and Oliveira [17], rays are emitted from the unknown pixel in different directions and the pixels that are not only on the ray paths but also on the boundary of known regions are collected as samples. On the other hand, He et al. [14] improved image matte quality by collecting a greater number of pixel samples. In their approaches, all pixels on the boundary of known regions are collected and a randomized search algorithm, rather than the brute force algorithm, is adopted to handle the large number of samples in pixel pair optimization. The sampling strategies mentioned above are known as local sampling strategies, as only the pixels on the boundary of known regions can be collected as samples. The approaches based on local sampling require a small amount of computing resources (most of them require hundreds to thousands PPE/UP [14], [17]). However, local sampling strategies may miss the true sample when the best pair of pixels is not located on the boundary, degrading the alpha matte quality significantly [15], [18]- [22].
Global sampling strategies were developed to alleviate missing true sample problem. Shahrian and Rajan [18], Shahrian et al. Shahrian2013 extended local sampling to global sampling by hierarchically clustering the pixels and selecting both the cluster centers and the pixels on the boundaries of known regions as samples. Feng et al. [20] used a pixel pair sampling objective function to select good pixel pairs from a candidate set obtained by k-means clustering. Distinct from previous sampling approaches, Johnson et al. [21] suggested that the unknown pixel color comprised of a set of unpaired pixel colors, and modeled natural image matting as a sparse coding problem of pixel features. Karacan et al. [22] considered pixel sampling as a sparse subset selection problem to increase the diversity of pixel samples. In their work, Kullback-Leibler divergence was employed to measure the feature distance among the cluster centers obtained by a superpixel approach, which provided a quantitative measurement of pixel sample dissimilarity. Huang et al. [15] cautioned that true sample may be missed due to: (1) conflicts among sampling criteria adopted; and (2) an incomplete sample space coused by using pixel clustering approaches. Thus they modeled pixel sampling for each unknown pixel as a multi-objective optimization problem and selected the Pareto optimal pixels as samples. In the multi-objective sampling, the pixel (rather than the superpixel) was used as the sampling unit. Global sampling strategies improve the diversity of pixel samples by expanding the sample space and designing sophisticated sampling strategy, while more computing resources are required (thousands to tens of thousands PPE/UP [15]) as the complexity of sampling strategy increases.
The main weakness of the sampling-based approach is its failure to adapt to the changes of available computing resources, because it uses a fixed amount of computing resources. Specifically, when the available computing resources are inadequate, the sampling-based approach cannot provide alpha mattes. On the other hand, it cannot utilize all of available computing resources when the computing resources are sufficient. Moreover, the issue of missing true sample cannot be completely eliminated in sampling-based approaches due to the complexity of pixel pair optimization [10].
Recently, several researchers [10], [23]- [25] demonstrated that pixel pairs can be optimized without sampling by using the evolutionary algorithm (a population-based metaheuristic optimization algorithm [26]), which can make use of all available computing resources by adjusting the iteration number of the evolutionary algorithm, and would in theory eliminate the risk of true sample omission. In these approaches (which are known as evolutionary-optimizationbased approaches), all PPO problems involved in an image are solved simultaneously by combining them into one large-scale PPO problem, as illustrated in Eq. (4).
where G(X ) is the fitness function for the large-scale PPO problem; g k (x k ) and x k denote the pixel pair evaluation function and the pixel pair for the k-th unknown pixel, respectively; N is the number of unknown pixels; and x denote the foreground-background pixel pair for the k-th unknown pixel. The evolutionary-optimization-based approach improves the performance of image matting by performing a global search with an evolutionary algorithm. However, the evolutionary-optimization-based approach requires extensive computing resources (hundreds of thousands PPE/UP [23]) to provide a high-quality alpha matte. Therefore, some attempts have been made to improve the search capability of evolutionary-optimization-based approaches. Cai et al. [23] suggested that correlations of unknown pixel colors can be used to improve the search capability of evolutionary-optimization-based approaches. In their work, a cooperative coevolution operator was embedded into differential evolution algorithm to group unknown pixels with similar colors and optimize the groups separately. In our previous study [10], we pointed out that each of the evaluation criteria involved in pixel pair evaluation can provide heuristic information for PPO. In our previous work, the heuristic information was used to support the optimization by transforming PPO problem into a multi-objective problem and solving it via a multi-objective evolutionary algorithm. Moreover, local smoothness prior was incorporated into the neighborhood grouping strategy such that pixel pairs can be optimized collaboratively, which significantly reduces demand for computing resources. Specifically, the pixel pairs regarding unknown pixels within a 9×9 region are considered as a group in the neighborhood grouping strategy. Half of pixel pairs in a group were selected and optimized, while the remaining pixel pairs adopted the optimization results shared by the selected pixel pairs. Our previous study demonstrated that the evolutionary-optimization-based approach can provide high-quality alpha mattes within thousands PPE/UP. However, evolutionary-optimization-based approaches still VOLUME 8, 2020 suffer from a major drawback that they cannot provide high-quality alpha mattes when computing resources are scarce (e.g. the available computing resources can only perform dozens PPE/UP).
Neither sampling-based approaches nor extant evolutionary-optimization-based approaches can provide satisfactory alpha mattes with different amounts of computing resources. There has been little discussion about adapting to the change of available computing resources in image matting tasks.
This issue motivated the present study, as a part of which we designed a multi-scale evolutionary pixel pair optimization framework named pyramid matting framework (PMF). PMF transforms the large-scale PPO problem into multiple PPO problems of different scales by image pyramid (a multi-scale image representation approach [27]), and solves them sequentially from small to large scales. The heuristic information found in a small-scale problem is propagated iteratively to the larger-scale one. The propagation generates a solution for the large-scale PPO problem using the heuristic information obtained in small-scale problems. Specifically, a pixel pair pertain to an unknown pixel obtained by solving a small-scale PPO problem is propagated to multiple spatial-related pixel pairs pertain to the unknown pixels within a local region in a larger-scale PPO problem. The heuristic information propagation provides the capability of computing resource adaptation for PMF, because it can not only generate alpha mattes according to a solution of the PPO problem at a small scale, but also make use of heuristic information obtained from solving small-scale PPO problems.
The remainder of this paper is structured as follows: Section II details the process of the proposed pyramid matting framework, while the experimental results are presented in Section III and Section VI concludes the paper.

II. PYRAMID MATTING FRAMEWORK
This section describes a computing-resource-adaptive multi-scale evolutionary pixel pair optimization framework named pyramid matting framework. Fig. 2 provides a graphical illustration of the proposed pyramid matting framework. The core PMF principle is based on searching for pixel pairs at different image scales and propagating the heuristic information from small to large scales. Local smoothness prior is utilized during heuristic information propagation. The heuristic information of each pixel pair in a small-scale image is propagated to multiple pixel pairs regarding the unknown pixels within a local region of a larger-scale image, as shown in Fig. 2. Moreover, with the heuristic information propagation, the PPO problem solution at the large scale can be generated according to the solution at any smaller scale, which provides the capability of adapting to the changes of available computing resources.
Algorithm 1 provides the pseudo-code of PMF, while the definitions of symbols used in PMF are shown in Table 1. PMF comprises two steps: (1) image matting pyramid construction; and (2) multi-scale pixel pair search with heuristic information propagation.
The details of image matting pyramid construction are provided in Subsection II-A, and the process adopted when searching for pixel pairs with heuristic information propagation is presented in Subsection II-B.

A. IMAGE MATTING PYRAMID CONSTRUCTION
The fist step in the PMF involves building multi-scale representation for both the image and the trimap (which we denote as image matting pyramid). Image pyramid [27] was adopted in PMF for two reasons.   Select a subset of pixel pairs n to be optimized on the n-th level of pyramids by using a selection criterion. n = n + 1 8: while pyramid construction stop criterion is not met 9: N p ← n 10: return P, N p (i.e. S ↓ T ()) to ensure that trimaps take appropriate values. The aim of image matting pyramid is to provide multi-scale representation for image matting. However, scaling images to a very small size may reduce level of detail considerably, as shown in Fig. 3(c). Thus, to avoid inclusion of overly small images into the image matting pyramid, the pyramid construction stop criteria were established. In the current implementation, the down-sampling loop terminates when one of the dimensions declines below 100 pixels. The image size set in the stop criterion is not a sensitive parameter, because downsample operator halves both the width and VOLUME 8, 2020 Algorithm 3 Heuristic Information Propagation (i.e. M(P(n + 1), X n+1 , P(n))) Input: P(n + 1), X n+1 and P(n). Output: X init n 1: for each pixel p in unknown regions of the n + 1-th level of the pyramid (i.e. P(n + 1)) do 2: (x 1 , x 2 ) ← X n+1 (p) 3: for each pixel q ∈ N (p) do 4: x 1 ←select a pixel from N (x 1 ) randomly 5: x 2 ←select a pixel from N (x 2 ) randomly 6: end for 8: end for 9: return X init n length of the images in each iteration. Other stop criteria can be used in image matting pyramid construction.

B. MULTI-SCALE PIXEL PAIR SEARCH WITH HEURISTIC INFORMATION PROPAGATION
Multi-scale pixel pair search plays an important role in PMF, as searches are conducted for all pixel pairs at each level of the image matting pyramid and the heuristic information is propagated across different-scale images. In multi-scale pixel pair search, the pixel pair search is performed level by level from top to bottom of the image matting pyramid, and heuristic information propagation and single-scale searching are performed alternately, as described in the iteration shown in Algorithm 1, lines 6-15. Notice that the PPO problem pertaining to the top level of the pyramid is optimized by the evolutionary optimizer before the iterative search (as shown in Algorithm 1, lines 5), which provides initial pixel pairs for heuristic information propagation.
Heuristic information propagation aims to propagate the heuristic information obtained by searching at the previous level of pyramid to the current level. However, the PPO problems corresponding to adjacent levels of the image pyramid change significantly in terms of decision vectors and decision space, although the corresponding images look highly similar. This poses the question of where the heuristic information should be propagated. The heuristic information propagation addresses this issue by determining correlations of pixel pairs among PPO problems at different scales. Algorithm 3 presents an implementation of heuristic information propagation. In heuristic information propagation, the neighbors of pixel pairs at the adjacent levels of the pyramid are defined using image matting pyramid. Here, unknown pixels will be used as an example to illustrate the definition of neighboring pixel. Those unknown pixels at current level of the pyramid used to calculate the average color of an unknown pixel in the previous level are defined as the neighbors of the unknown pixel. The neighborhood relations of pixels at different scales can be extended to pixel pairs, since each pixel pair corresponds to a pixel in unknown regions. Definitions of neighbors of a foreground or background pixel can be given by analogy.
Decision vector transformation and decision space transformation are involved in heuristic information propagation. Decision vector transformation utilizes local smoothness prior in unknown regions. The pixel pair decision vector at the previous level of the pyramid is transformed to that at the current level (which corresponds to a larger-scale PPO problem) according to pixel pair neighborhood relations in the decision vector transformation process. However, pixel pairs obtained from decision vector transformation cannot be used directly at the current level of the pyramid due to the change in the decision space. Search space transformation was thus designed to transform foreground pixels and background pixels from a small-scale PPO problem to a larger-scale PPO problem. Either the foreground pixel or the background pixel of a pixel pair obtained by decision vector transformation may correspond to more than one neighboring pixels in the current level of the pyramid. One foreground pixel and one background pixel were determined for each pixel pair in a larger-scale PPO problem by applying random selection on neighbors separately.
Single-scale search is performed after heuristic information propagation for each iteration (as described in Algorithm 1, lines 10-12), in which heuristic information that has been propagated from the previous level of the pyramid is used through pixel pair selection and pixel pair evolutionary optimization. Some of the pixel pairs in the initial decision vector obtained by heuristic information propagation are of high quality and can provide accurate alpha value estimation due to the local smoothness prior. It is not necessary to apply evolutionary optimization on high-quality pixel pairs. Consequently, only half of pixel pairs with low fitness are selected from the initial decision vector to be optimized in pixel pair selection process (which implements the selection criterion in Algorithm 1, line 11). The pixel pair selection avoid redundant search and can therefore improve the search efficiency. Pixel pair evolutionary optimization is subsequently applied to each selected pixel pair. The pixel pair in the initial decision vector is provided as an initial individual to the evolutionary optimizer to use the heuristic information propagated from the small-scale PPO problems.

III. EXPERIMENTAL RESULTS
In this section, findings yielded by three experiments are reported to evaluate the presented pyramid matting framework with different amounts of computing resources.

A. EXPERIMENTAL DESIGN
Three representative cases were considered in the experiments, which correspond to the cases where image matting approaches can be run for hours, minutes and seconds in a workstation computer, respectively. In case one, the maximum number of PPE/UP was set to 5000 (which is consistent with the setting in [10]) to evaluate the performance of PMF with sufficient computing resources. The maximum number of PPE/UP was set to 50 in case two, which corresponds to the case where image matting approaches are required to provide an alpha matte within minutes on a workstation computer. The maximum number of PPE/UP was further reduced to five in case three to demonstrate the performance of involved approaches with very few computing resources. The first experiment was conducted to verify the PMF effectiveness with sufficient computing resources by comparing the performance of evolutionary-optimization-based approaches with and without PMF in case one. The second experiment shows the performance of PMF-based approach in case two and case three where limited computing resources are available. In the last experiment, a visual comparison was performed among the alpha mattes obtained by the PMF-based approach and that yielded by the state-of-the-art image matting approaches based on pixel pair optimization.
All three experiments were carried out on a popular image matting benchmark dataset [28], which provides 27 color images with their ground-truth alpha mattes. Two types of trimaps, including the trimaps with small unknown regions and that with large unknown regions, were used in the experiments. The ground-truth alpha mattes were used to evaluate the performance of the involved approaches. Notice that all involved approaches were performed without pre-processing and post-processing processes in the experiments to provide a fair comparison, because these processes may affect the quality of alpha mattes [29]. The pixel pair evaluation function presented in the literature [10] was used in the pixel pair optimization for all involved evolutionary-optimizationbased approaches. A widely used performance metric named mean square error (MSE) was used to measure the error between an estimated alpha matte and the ground-truth matte quantitatively, which can be written as: where ε denotes the MSE of an estimated alpha matte. α p andα p denote the alpha values at pixel p in the estimated and the ground-truth alpha matte, respectively. A small observed value of MSE indicates a high-quality alpha matte. All experiments were conducted on a workstation computer with an Intel Core i7 3.6-GHz CPU and 32 GB of memory. The involved approaches were implemented in MATLAB.

B. IMAGE MATTING WITH SUFFICIENT COMPUTING RESOURCES
The aim of the first experiment was to ascertain whether PMF can provide high-quality alpha mattes when computing resources are sufficient. A comparison between evolutionaryoptimization-based approaches with and without PMF was conducted using the setting of case one in this experiment. Two image matting approaches were implemented within PMF by taking a state-of-the-art evolutionary-optimizationbased approach named multiobjective evolutionary algorithm based on multicriteria decomposition (MOEAMCD) [10] and a popular large-scale evolutionary optimizer named competitive swarm optimization (CSO) [30] as evolutionary optimizers, respectively. The parameters involved in the evolutionary optimizers were adopted from pertinent literature [10], [30], respectively. The resulting PMF-based approaches are called MEOAMCD-PMF and CSO-PMF, respectively. Consequently, the original MOEAMCD and CSO were used as a baseline in the comparison. All involved approaches were run 30 times on each image separately to allow for statistical analysis of their performance. Table 2 and Table 3 provide the average value and standard deviation of MSE obtained by the evolutionary-optimizationbased approaches with and without PMF. Table 2 shows the results yielded by using the trimap with small unknown regions, while the results of using the trimap with large unknown regions are shown in Table 3. The average values of MSE obtained by PMF-based approaches are smaller than those obtained by the original approaches in almost all the cases, as shown in Table 2 and Table 3. MOEAMCD-PMF outperforms MOEAMCD on 23 out of 27 images in terms of average MSE in the comparison involving the trimap with small unknown regions, as well as on 21 out of 27 images in the comparison involving the trimap with large unknown regions. Moreover, CSO-PMF provides alpha mattes with lower average MSE than CSO on all 27 images in both comparisons. These experimental results indicate that the quality of alpha mattes obtained by the involved approaches was improved by adopting PMF in the case with sufficient computing resources. The observed decline in MSE can be attributed to the search capability improvement through heuristic information propagation, which provides useful heuristic information across the multi-scale pixel pair search. With the propagated heuristic information, the search capability of evolutionary-optimization-based approaches can be improved when the computing resources are sufficient.

C. IMAGE MATTING WITH LIMITED COMPUTING RESOURCES
The goal of the second experiment was to verify if PMF can provide high-quality alpha mattes when computing resources are limited. Case two and case three were considered in this experiment. Two PMF-based approaches (MOEAMCD-PMF and CSO-PMF) were involved in this experiment, and the approach with the closest performance to the PMF-based method in the first experiment, MOEAMCD, was used as a baseline. This experiment was conducted using the trimap with small unknown regions in the dataset [28]. The involved approaches were run 30 times to facilitate a statistical analysis of their performance. Table 4 shows the average MSE of 27 images obtained by PMF-based approaches and MOEAMCD [10] in case two and case three. Notice that PMF-based approach works well in both cases, whereas MOEAMCD failed in case three, since computational resource was insufficient for MOEAMCD in this case. 40 PPE/UP were performed in the neighborhood grouping strategy employed in MOEAMCD, but only five PPE/UP could be evaluated in case three. Image matting performance comparison between evolutionary-optimization-based approaches with and without pyramid matting framework in terms of the average value and standard deviation of MSE for 27 images from the benchmark dataset [28] using trimap with small unknown regions. The maximum number of pixel pair evaluations per unknown pixel was set to 5000. The bold numbers represent the better MSE in the comparison.

TABLE 3.
Image matting performance comparison between evolutionary-optimization-based approaches with and without pyramid matting framework in terms of the average value and standard deviation of MSE for 27 images from the benchmark dataset [28] using trimap with large unknown regions. The maximum number of pixel pair evaluations per unknown pixel was set to 5000. The bold numbers represent the better MSE in the comparison.
MOEAMCD-PMF provided alpha mattes with lower average MSE than MOEAMCD on 15 out of 27 images in case one. These results indicate that the PMF can adapt to the changes in computing resources and provide alpha mattes with different amounts of computing resources. This success can be attributed to the multi-scale pixel pair search with heuristic  [28] obtained by PMF-based approaches (MOEAMCD-PMF and CSO-PMF) and a state-of-the-art evolutionary-optimization-based image matting approaches (MOEAMCD) [10] using trimap with small unknown regions. The maximum number of PPE/UP was set to 50 and 5 in case one and case two, respectively. The bold numbers represent the best MSE in the comparison. information propagation, which prioritizes small-scale PPO problems and transforms a solution of a small-scale PPO problem to the corresponding solution of the large-scale PPO problem when computing resources are insufficient. When the computing resources are inadequate for optimizing all PPO problems of different scales, heuristic information propagation generates alpha mattes by transforming the best known solution for a small-scale PPO problem to the solution for the large-scale PPO problem. Heuristic information obtained from the small-scale problem is effectively used during the propagation, which ensures the quality of the generated solution. Although MOEAMCD-PMF still outperforms MOEAMCD on 15 out of 27 images, its performance advantage of MOEAMCD-PMF over MOEAMCD declines when restrictions are imposed on computing resources. Similar findings were obtained in the comparison between MOEAMCD-PMF and CSO-PMF in case three, as shown in Table 4. These results can be explained by the imbalance of computing resource between the pixel pair search and the heuristic information propagation. When the available computing resources are limited, the neighborhood grouping strategy of MOEAMCD may take up most of the resources, making pixel pair search challenging. Although PMF provides heuristic information across the multi-scale pixel pair search, MOEAMCD-PMF exacerbates the imbalance reducing the quality of identified pixel pairs, because additional VOLUME 8, 2020 FIGURE 5. Visual comparison of alpha mattes obtained by MOEAMCD-PMF, MOEAMCD [10], KL divergence matting [22] PDMS matting [15] and CCDE matting [23]. (a) Input image in which the red line and the blue line denote the boundary of known foreground regions and background regions respectively. (b) Zoomed-in region. (c) Ground-truth alpha matte. (d) Alpha matte obtained by MOEAMCD-PMF. (e) Alpha matte obtained by MOEAMCD [10]. (f) Alpha matte obtained by KL-divergence matting [22]. (g) Alpha matte obtained by PDMS matting [15]. (h) Alpha matte obtained by CCDE matting [23]. Arrows indicate regions of low-quality mattes.
computing resources are required in PMF resulting that fewer computing resources can be allocated to pixel pair search. CSO-PMF does not involve neighborhood grouping strategy, as a result, more computing resources are allocated to pixel pair search redressing the computing resource imbalance. Therefore, CSO-PMF outperforms MOEAMCD-PMF in 10 out of 27 cases in case three. Fig. 4 shows a visual comparison of alpha mattes obtained by MOEAMCD [10], MOEAMCD-PMF and CSO-PMF in case two. Both MOEAMCD-PMF and CSO-PMF provide alpha mattes with sharp boundaries and less noise in case two. The results confirms that PMF can adapt to the change of available computing resources, and PMF-based approaches provide acceptable alpha mattes even when the available computing resources are scarce. However, the alpha mattes obtained by PMF-based approaches with few computing resources degrade when the local smoothness assumption is not satisfied (e.g. hole), as shown in the last row of Fig. 4.

D. VISUAL EVALUATION
The third experiment was conducted to provide a visual comparison of alpha mattes among the PMF-based approach and other state-of-the-art PPO-based approaches. MOEAMCD-PMF, two state-of-the-art sampling-based image matting approaches (pixel-level discrete multiobjective sampling (PDMS) matting [15] and KL-Divergence matting [22]) and two recently proposed evolutionaryoptimization-based image matting approaches, including MOEAMCD [10] and cooperative coevolution differential evolution matting (CCDE) [23], were involved in this experiment. The maximum number of PPE/UP for the involved evolutionary algorithms was set to 5000 in this experiment, considering thousands PPE/UP were performed in the involved sampling-based image matting approaches [15].
As shown in Fig. 5, the evolutionary approaches based on local smoothness prior (MOEAMCD-PMF and MOEAMCD) provide high-quality alpha mattes, while the evolutionary approaches without local smoothness prior (CCDE) do not. This finding indicates that the local smoothness prior plays an important role in pixel pair optimization. Moreover, the alpha matte obtained by MOEAMCD-PMF is visually superior to that yielded by MOEAMCD, which confirms that PMF improves the search capability of evolutionary-optimization-based approaches and provides a feasible multi-scale evolutionary pixel pair optimization framework to efficiently solve PPO problem. MOEAMCD-PMF also outperforms the state-of-theart sampling-based approaches as the resulting alpha mattes provide fine detail and generate less noise.

IV. CONCLUSION
This paper presents a multi-scale evolutionary pixel pair optimization framework (pyramid matting framework) for image matting. It provides a feasible way of adapting to the change of available computing resources for the image matting approaches based on evolutionary optimization. PMF first transforms the large-scale PPO problem to multiple PPO problems at different scales by constructing image pyramid for the input image and trimap. The pixel pair search is subsequently applied level by level, from the top to the bottom of the image matting pyramid. PMF propagates the heuristic information of pixel pairs found in the PPO problem at a small scale to that at a higher scale. Each optimized pixel pair in a small-scale image is propagated to multiple pixel pairs regarding the unknown pixels within a local region of a larger-scale image. A reasonable solution of the large-scale PPO problem can be generated with the heuristic information propagation according to the PPO problem solution at small scales. We have shown that PMF can adapt to the change of available computing resources and have demonstrated that evolutionary-optimization-based approaches with PMF not only provide high-quality alpha mattes when computing resources are sufficient but also provide acceptable alpha mattes even under computing resource limitations. However, PMF-based approaches cannot provide high-quality alpha mattes when unsmooth regions are involved and computing resources are scarce. Further work may include adaptive computing resource allocation between the pixel pair search and the heuristic information propagation and scene-aware pixel pair optimization strategy for unsmooth regions.