N-White Balancing: White Balancing for Multiple Illuminants Including Non-Uniform Illumination

In this paper, we propose a novel white balance adjustment for multi-illuminant scenes, called “N-white balancing,” in which $N$ source white points are mapped into a ground truth one. Most white balance adjustments focus on adjusting single-illuminant scenes. Several state-of-the-art methods for adjusting multi-illuminant scenes have been proposed, but they need to know the number of segments or the number of light sources in advance. Multi-color balance adjustments have been investigated to improve the performance of white balancing, but they also suffer from color distortion due to the rank deficiency problem. In contrast, the proposed method, N-white balancing, can correct multi-illuminant scenes even when we do not know the exact number of segments or light sources in a scene. In an experiment, the proposed method was demonstrated to outperform state-of-the-art methods under various illumination conditions such as single and multiple illuminants including non-uniform light sources.


I. INTRODUCTION
Image segmentation and object recognition are required to decompose an image into meaningful regions. A typical approach to this problem assigns a single class to each pixel in an image. However, such hard segmentation is far from ideal when the distinction between meaningful regions is ambiguous, such as in the cases of objects with motion blur or color distortion caused by illumination or in the case of image enhancement [1], [2], [3], [4], [5]. Accordingly, we aim to reduce lighting effects in an image.
A change in illumination affects the pixel values of an image taken with an RGB digital camera because the values are determined by spectral information. In the human visual system, it is well known that illumination changes (i.e., lighting effects) are reduced, and this ability keeps the entire color perception of a scene constant. In contrast, since cameras do not essentially have this ability, white balancing is applied to images [6], [7], [8].
The associate editor coordinating the review of this manuscript and approving it for publication was Gustavo Callico .
Most traditional white balancing attempts to remove lighting effects on the assumption that a scene is captured under a single light source. Such white balancing for single illuminants (i.e., single-illuminant WB) requires two steps: estimating a source white point and mapping the estimated source white point into a ground truth white point without lighting effects, where ''source white point'' is a set of tristimulus values that represents the color of a white region when a light source strikes the white region, ''white region'' is a region that reflects all of the light, and ''ground truth white point'' is a source white point measured under an ideal light source such as the CIE standard illuminant D65 [9]. Accordingly, a source white point represents the color of illumination. In general, a source white point is calculated as a representative (e.g., the mean or median) value from the pixel values of the white region, and it is denoted by a set of tristimulus values such as (X, Y, Z) in the XYZ color space [10]. The phrase ''source white point'' is sometimes referred to as ''illuminant.'' Many studies have focused on automatically estimating a source white point in an image [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ [28], [29], [30], [31], [32], [33], [34]. Also, some methods automatically select the most appropriate algorithm for illuminant estimation depending on the content of individual images [35], [36], [37]. However, single-illuminant WB cannot reduce lighting effects under multiple illuminants. Mixed and non-uniform light sources are examples of multiple illuminants.
Multi-color balance adjustments [8], [38], [39] have been proposed to consider correcting various colors. In these methods, multiple colors are used for designing a matrix that maps multiple colors into ground truth colors. However, multi-color balancing may cause the rank deficiency problem to occur as a non-diagonal matrix is used, unlike white balancing [40].
To address this problem, state-of-the-art white balance adjustments for mixed light sources (i.e., multi-illuminant WB) have been proposed [25], [41], [42], [43], [44], [45]. By using a segment-wise estimation of a source white point, these methods can adjust lighting effects under single or mixed light sources. However, in some cases, the number of segments or the number of light sources needs to be known in advance [46]. Wrong assumptions result in color distortion in an image [45]. Moreover, these methods consider multiple illuminants as mixed light sources, so non-uniform illumination such as shading cannot effectively be adjusted. Barnard et al.'s method [47] considers non-uniform illumination. However, it assumes that the illuminant transition is smooth. Several approaches [26], [27], [48], [49] based on the human vision system (HVS) have been proposed, and some of them [48], [49] can be applied for adjusting multi-illuminant scenes. Also, methods based on machine learning [46], [50], [51], [52] have been studied. However, both HVS-and learning-based methods have problems such that any previously studied illuminant estimation algorithm cannot be used. Accordingly, in this paper, we propose a novel white balancing called ''N-white balancing'' for adjusting images taken under single and multiple illuminants. N is the number of source white points, not the number of light sources. Under a general single-illuminant scene, N source white points have almost the same set of values. In contrast, under a multi-illuminant scene, source white points calculated from N different positions are generally different. The proposed method aims to simply adjust N source white points to corresponding ground truth ones. Hence, the method can reduce lighting effects under various illumination conditions even when N is a larger number than the number of light sources in a scene. Therefore, N-white balancing relieves us of needing to accurately select such parameters. Additionally, color distortion due to a rank deficient matrix is never caused in N-white balancing because a diagonal matrix is used.
In experiments, the proposed method was compared with single-illuminant WB, multi-color balancing, and multi-illuminant WB on the basis of the reproduction angular error [53]. The rest of this paper is organized as follows. In Section II, we review related work. In Section III, we present our N-white balancing for single and multiple illuminants. In Section IV, we evaluate the effectiveness of the proposed method. In Section V, we conclude our research.

II. RELATED WORK
We summarize conventional methods for color constancy to clearly show problems with these methods. Conventional methods are categorized into three types: singleilluminant WB, multi-color balancing, and multi-illuminant WB. In Table 1, the differences among the color balance adjustments are summarized in terms of the number of light sources and the selection of source points. The proposed method is a type of multi-illuminant WB.

A. LIGHTING EFFECTS ON PIXEL VALUES
On the basis of the Lambertian model [54], the pixel values of an image taken with an RGB digital camera are determined by using three elements: spectra of illumination E(λ), spectral reflectance of objects S(λ), and camera spectral sensitivity R C for color C ∈ {R, G, B}, where λ spans the visible spectrum in the range of [400, 720]. A pixel value P RGB = (P R , P G , P B ) in the camera RGB color space is given by Eq. (1) means that a change in illumination E(λ) affects the pixel values in an image. In the human visual system, the changes (i.e., lighting effects) are excluded, and the overall color perception is constant regardless of illumination differences, known as color constancy. To mimic this human ability as a computer vision task, white balancing (WB) is typically performed as a color adjustment technique.

B. WHITE BALANCING FOR SINGLE ILLUMINANT
Under a single light source, single-illuminant WB can reduce lighting effects on colors in an image. A two-step procedure is required to perform single-illuminant WB: 1) estimate a source white point, and 2) map the estimated source white point into a ground truth white point. Many studies on color constancy have focused on the first step (i.e., source white point estimation) [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34]. Methods for estimating a source white point are categorized into three types. The first type is methods that need the locations of white regions to calculate a source white point from white regions. Yang et al. proposed a method for estimating a source white point under the framework of the Grey-Pixels adjustment [25], and the method belongs to this first type. The second type is methods that determine a source white point from pixels satisfying a specific condition [11], [12], [15], [19], [24], [26], [27], [30], [55]. For example, the White-Patch algorithm assumes that the maximum pixel values in RGB channels represent a perfect reflectance of the light source [11]. These methods do not require that the locations of white regions be estimated, unlike the estimation method of the Grey-Pixels adjustment. The third type is methods that determine a source white point by using machine learning [13], [14], [16], [17], [20], [21], [22], [23], [28], [29], [32], [33], [34], [56], [57]. These approaches can also compute a source white point without estimating the locations of white regions. Therefore, the estimation methods of the second and third types enable us to estimate a source white point without white regions. Alternatively, they approximate the pixel values of white regions by using other pixel values in an image such as the maximum pixel value in the image. Performing both the first and the second steps is called ''auto white balance adjustment'' because white balance adjustment can be fully automated. In contrast, we may decide a source white point by manually pointing out a white region with remaining lighting effects under some conditions, which is called ''manual white balance adjustment.'' Under these conditions, the error in illuminant estimation is ignored.
After obtaining illuminant information automatically or manually, the second step is performed by where P XYZ = (X P , Y P , Z P ) is a pixel value of an image in the XYZ color space [10], and P WB = (X WB , Y WB , Z WB ) is that of a white-balanced image [58]. M WB in (2) is given as M A with a size of 3 × 3 is decided in accordance with an assumed chromatic adaptation transform [58]. (ρ S , γ S , β S ) and (ρ D , γ D , β D ) are calculated from an obtained source white point of a light source (X S , Y S , Z S ) in an input image and a ground truth white point ( Using the 3 × 3-identity matrix as M A indicates that white balancing is performed in the XYZ color space. Otherwise, von Kries's [59], [60] and Bradford's [61] chromatic adaptation transforms, which were also proposed for reducing lighting effects on all colors under the framework of white balancing [62], can be used. For example, under the use of Bradford's model, M A is given as Single-illuminant WB is a technique that maps a source white points under a single light source into a ground truth one as in (3). However, most techniques for single-illuminant WB do not consider the adjusting of spatially varying colors caused under mixed or non-uniform light sources. Accordingly, it suffers from such multiple illuminants in terms of color constancy. To address this problem, state-of-theart white balance adjustments that assume multi-illuminant scenes have been proposed, as described in Section II-D.

C. MULTI-COLOR BALANCING
Multi-color balance adjustments [8], [38] have been proposed to consider adjusting multiple colors under single illuminants, while single-illuminant WB considers adjusting a single color (i.e., white). In these methods, multiple colors are used for designing a non-diagonal matrix that maps the multiple colors into ground truth colors. Hence, source white points under multiple illuminants may also be adjusted by using multicolor balancing. However, when we include multiple source white points as multiple colors, the non-diagonal matrix will be rank deficient. To avoid the rank deficiency problem, various chromatic colors in addition to white should be selected for the matrix design. However, automatic estimation of chromatic colors under illumination is difficult, unlike auto white balance adjustments.

D. WHITE BALANCING FOR MULTIPLE ILLUMINANT
Only a handful of methods [41], [42], [43], [44], [45], [46], [47], [50], [51], [52], [63], [64] have been proposed to correct multiple source white points under mixed light sources. Because multi-illuminant WB is a type of white balancing, illuminant estimation may be used; moreover, the rank deficiency problem is never caused. Early work in [42], [43], and [44] uses a segment-wise illuminant estimation that divides images into small segments. In these methods, estimated illuminants are clustered by using K-means clustering. However, it is difficult to exactly set the parameters of clustering in general scenes. Yang et al. proposed the Grey-Pixels adjustment for multiple illuminants [25] in addition to that for single ones. However, the Grey-Pixels adjustment used for single illuminants is different from that used for multiple illuminants, so this method is not applicable if it is difficult to distinguish single-illuminant scenes from multi-illuminant ones. Moreover, the method has to use the illuminant estimation algorithm proposed for the Grey-Pixels adjustment. Hussain et al.'s method [45] splits an input image into multiple segments using the K-means ++ algorithm to improve the accuracy of the color adjustment. However, the authors reported that the number of segments significantly affected the accuracy [45]. Therefore, the conventional multi-illuminant WB requires the exact number of segments or light sources in advance [46]. Wrong assumptions may cause incorrect color correction in the methods. Additionally, non-uniform light sources (e.g., shading) are not considered in these methods.  set the number of clustering depending on the number of light sources. However, a more accurate mimic of realistic retinal mechanisms is required to improve the color correction performance, which has not been achieved yet [48]. Moreover, previously studied illuminant estimation algorithms cannot be used in these HVS-based methods since the HVS does not have the ability to estimate source white points directly.
A number of neural network-based methods have also been studied for multi-illuminant WB [46], [51], [52]. The performance of these methods is almost the same as that of multi-illuminant WB without a neural network. However, they suffer from two difficulties compared with multi-illuminant WB without a neural network: needing to collect a massive amount of images captured under multiple illuminants as training datasets and needing to implement a neural network in a camera system [46], [48].
Accordingly, this paper presents ''N-white balancing'' as a novel multi-illuminant WB. In the proposed method, N source white points are mapped into a ground truth one. Hence, the method does not need illuminant clustering. Also, N source white points can be manually decided or automatically estimated with every type of illuminant estimation algorithm. The proposed method focuses on simply correcting N source white points, for which we do not need to choose the correct number of light sources or segments as N , but some other adjustments for multi-illuminant scenes need it. Moreover, the method does not include complex tasks such as an accurate mimic of realistic retinal mechanisms, compared with methods based on the HVS [48], [49]. Therefore, the method can adjust images with no distinction between single-and multi-illuminant scenes, unlike the Grey-Pixels adjustment.

A. OVERVIEW OF PROPOSED METHOD
Even when the color of an object is white, it may be different from the white under a light source. If we select several source white points in a general single-illuminant scene, they may have almost the same set of values in general (see Fig. 1 (a)). In contrast, in a multi-illuminant scene such as one with mixed and non-uniform light sources, the values of source white points calculated from white regions in spatially different positions are generally different. (see Figs. 1 (b)-(c)). The proposed method individually adjusts N source white points to a ground truth one. Hence, the method can stably reduce lighting effects even when N is a larger number than the number of light sources in a scene. Colors among N source white points are also corrected by using N weights.

B. N-WHITE BALANCING
The proposed method is carried out by using N source white points, and they are combined with weights. In the method, a pixel value P XYZ = (X P , Y P , Z P ) is adjusted to P WB = (X WB , Y WB , Z WB ) as In general, ground truth white points have the same value as (6) is designed in a similar way to white balancing: where , 2, · · · , N }) in the XYZ color space, and k m is a weight of S m and G m . In a manner like the conventional white balancing, M A is used for accurately adjusting colors other than achromatic colors. The 3 × 3-identity matrix is used as M A when the proposed method is performed in the XYZ color space. von Kries's model or Bradford's one may also be used as M A . As mentioned in Section II-B, the effectiveness of M A has been confirmed for single-illuminant scenes. In this paper, we verify the effectiveness of M A for multi-illuminant scenes in Section IV. When the spatial pixel coordinate of P XYZ is closer to that of source white point S m than the other ones, S m should more contribute to adjusting P XYZ than the rest of the source white points. Hence, to measure the distance between the coordinates of P XYZ and S m , the Euclidean distance is calculated as where (x Sm , y Sm ) is a pair of coordinates of S m , and (x P , y P ) is that of P XYZ . A smaller d m means that (x P , y P ) is closer to (x Sm , y Sm ). Because k m in (8) should be larger under a smaller d m , the inverse proportion to d m is calculated as To reduce the total value of weights to 1 in (8), k m is given as 89054 VOLUME 10, 2022 Note that, in (11), k m will be infinite if the pixel coordinate of input pixel P XYZ is equal to that of S m (i.e., d m = 0). In this case, let k m be a value of 1, and let the other weights be a value of zero. In the proposed method, the use of the Euclidean distance derives from the Grey-Pixels adjustment for multi-illumination [25]. However, the calculations are different from the Grey-Pixels adjustment as in (10)- (11), and there is no parameter such as the weighting sensitivity that Grey-Pixels has.

C. PROCEDURE OF N-WHITE BALANCING
To perform the proposed method, three sets of data are prepared: a ground truth white point , 2, · · · , N }), and the pixel coordinates (x Sm , y Sm ) corresponding to the N source white points. The procedure of the proposed method is carried out as below.
1) Decide a ground truth white points as G.
2) Prepare white regions in various locations of a camera frame to compute S m . 3) Obtain the pixel coordinates (x Sm , y Sm ) of S m . 4) Apply N-white balancing, following Section III-B. N-white balancing is a kind of multi-illuminant WB, so illuminant estimation algorithms can be used instead of preparing white regions in step 2). For example, to automatically estimate N source white points, block segmentation is applied to an image, and a block-wise illuminant estimation algorithm can be used (see Fig. 2 (a)). In this paper, the cosine similarity between S m and the pixels of each block is calculated, and the coordinates of the closest pixel to S m are decided as (x Sm , y Sm ) (see Fig. 2 (b)).
In the proposed method, the locations of N source white points are used, which are decided from estimated source white points and the pixels of each block. The method does not need the locations of white regions, unlike the Grey-Pixels adjustment [25]. As discussed in Section II, this is because some methods that estimate a source white point without white regions have been proposed, and these methods can be applied to the illuminant estimation part of the proposed method.

D. PROPERTIES OF N-WHITE BALANCING
The properties that the proposed N-white balancing has are summarized here. (a) The rank deficiency problem is never caused since a diagonal matrix is used as in (7), unlike multi-color balancing. (b) N does not need to correspond to the exact number of light sources or segments. (c) Source white points may be selected manually in addition to automatically estimating them. (d) Any previously studied algorithm can be used for the illuminant estimation part of N-white balancing. In N-white balancing, we apply block segmentation to decide N source white points as described in Section III-C, so the proposed method is effective in adjusting multi-illuminant scenes including non-uniform illumination. In addition, when using a fixed-point camera, source white points may manually be decided, so that the performance is improved and the process is simplified.

IV. EXPERIMENTS
We conducted experiments to confirm the effectiveness of the proposed method.

A. EXPERIMENTAL SETUP
In the first experiment, we used five images taken under various illumination conditions such as single, mixed, and non-uniform light sources, where each image included a color rendition chart (see Fig. 3). In this experiment, source white points were manually decided from the white regions in the color rendition charts or automatically estimated by using an illuminant estimation algorithm. We also confirmed that the performance of N-white balancing was maintained regardless of N . Note that the proposed method and single-illuminant WB were combined with Bradford's model [61]. The performance of each method was evaluated by using the reproduction angular error [53] between a mean-pixel vector of an adjusted object's region P and that of the corresponding ground truth one Q. The reproduction error between P and Q is given by VOLUME 10, 2022 In the second experiment, we used the two-illuminant dataset [43], which consists of 58 laboratory images taken under close-to-ideal conditions and 20 real-world images. Similarly to the previous experiment with color rendition charts, we demonstrated that the proposed method was also effective in adjusting images taken in general scenes. In this experiment, our error metric per image was the mean pixel-wise angular error between estimated illuminant i and the corresponding ground truth one j, given as where n p is the number of pixels in an image.
In the third experiment, we used two state-of-the-art datasets: the LSMI dataset [65] and Afifi et al.'s mixedilluminant test set [46]. The LSMI dataset consists of natural scenes in the real world, and it has three subsets named ''galaxy,'' ''nikon,'' and ''sony.'' In this experiment, we used 1,124 two-illuminant scenes in the galaxy subset, and the performance of each method was evaluated by using the mean pixel-wise angular error as in (13). Afifi et al.'s mixedilluminant test set includes 150 synthetic images rendered by 3Ds Max. In this dataset, the reproduction angular error as in (12) was used to evaluate each method.

B. EXPERIMENTAL RESULTS WITH MANUALLY DECIDED SOURCE WHITE POINTS
The effectiveness of N-white balancing was confirmed under the use of manually decided source white points.

1) EVALUATION OF SINGLE AND MULTIPLE ILLUMINANTS
In this experiment, we first confirmed that the proposed method is effective for correcting both single-and multipleilluminant scenes. By using source white points decided from the white regions in a color rendition chart, the proposed method was compared with single-illuminant WB and multicolor balancing [8]. Figure 4 shows the adjusted images for Figs. 3 (a)-(d). The heat map below each image indicates the reproduction angular error for every color patch, where P in (12) is the representative pixel value of one of the color patches in an adjusted image for Figs. 3 (a)-(d), and Q is that for the ground truth image (see Fig. 3 (e)). The mean value of the pixel values in a color region (i.e., color patch) was used as a representative value. Also, the mean value (Mean) and the standard variation (Std) of all color patches are shown next to each heat map. Note that patches whose color is pure black were excluded from the calculation of Mean and Std because the metric cannot precisely measure errors against low surface reflectances. Source white points were decided from white regions represented as the green rectangle in the figures.
As shown in Fig. 4, the proposed method reduced both single-and multi-illuminant effects, and it had the lowest mean error in all situations. In contrast, the multi-illuminant scenes were not adjusted by single-illuminant WB, although the single-illuminant scene was corrected as well as the proposed method. Also, multi-color balancing caused the rank deficiency problem to occur, and the chromatic colors were distorted. Figure 5 shows the relationship between mean errors and N (i.e., the number of source white points). In Figs. 3 (a) and (b), the scene was taken under a single light source and two different light sources, respectively. As shown in Fig. 5 (b), when N was below the number of light sources in the scene, N-white balancing could not correctly adjust the images. However, when N was larger than the number of light sources, the performance of N-white balancing was maintained. Therefore, N does not need to correspond to the exact number of light sources.

C. EXPERIMENTAL RESULTS WITH AUTOMATICALLY ESTIMATED SOURCE WHITE POINTS
In this section, the effectiveness of N-white balancing is demonstrated under the use of automatically estimated source white points.

1) EVALUATION OF SINGLE AND MULTIPLE ILLUMINANTS
First, the proposed method with manually decided source white points was compared with automatically estimated ones. In this experiment, the White-Patch algorithm [11] was used for estimating source white points. Also, a block size of 3 × 3 was decided for the proposed method as the performance of the method is maintained when N is larger than the number of light sources in a scene (see Section III-C). Figure 6 shows a comparison between manual white balancing and automatic white balancing. The performance of color correction did not significantly differ between using manually decided source white points and using automatically estimated ones in these images. Also, the use of source white points manually selected from white regions (i.e., manual white balance adjustments) slightly improved the performance of the color adjustments. Therefore, the effectiveness of property (III-D) in Section III-D was confirmed, while the conventional multi-illuminant WB can only be used for auto white balance adjustments.
In following experiments, we confirmed that the proposed method is effective as an automatic white balance adjustment in adjusting images under single and multiple illuminants. In this experiment, the proposed method, single-illuminant WB, and multi-illuminant WB [25], [42], [45], [46], [48] were tested by using the White-Patch algorithm [11], which is a typical illuminant estimation method. The number of illuminants was set to two in Gijsenij et al.'s and Yang et al.'s method [25], [42], and the number of segments was set to four in Hussain et al.'s method [45], and these settings were recommended by the respective authors. Table 2 shows the Mean and Std of the reproduction angular errors for the white balance adjustments. The proposed method outperformed the conventional singleand multi-illuminant WB in terms of mean errors. In addition, the effectiveness of single-illuminant WB was confirmed for the single-illuminant scene, which showed the same trend as that seen in Section IV-B1. The multi-illuminant WB without a neural network [25], [42], [45], [48] had unstable performance depending on the illumination conditions, while the proposed method stably adjusted images under such conditions. The multi-illuminant WB based on a neural network by Afifi et al. [46] suppressed single-and multiilluminant effects; however, the Mean and Std of the angular errors were higher than those for the proposed method.
Therefore, the effectiveness of the proposed method for single and multiple illuminants including non-uniform light sources was confirmed.

2) INFLUENCE OF USING SAME N FOR VARIOUS ILLUMINATION CONDITIONS
As mentioned, in the proposed method, N does not need to correspond to the exact number of light sources or the exact number of scene segments. In this section, the proposed method was compared with multi-illuminant WB [42], [45] without a neural network by changing parameters. VOLUME 10, 2022    Fig. 3 (a). Two different parameters were tested in Gijsenij et al.'s method [42] as shown in Figs. 7 (b) and (c); the number of illuminants was modified from two to one because only one light source illuminated the scene. Figure 8 shows adjusted images for Fig. 3 (b). Two different parameters were tested in Hussain et al.'s method [45] as shown in Figs. 8 (b) and (c); the number of segments was reduced from four to two because there were two light sources. Comparing Figs. 7 (b) and 8 with Figs. 7 (c) and 8 (c), respectively, the conventional multi-illuminant WB [42], [45] may require the exact number of light sources or segments, and accurately adjusting the parameters significantly contributed to improving the performance. In contrast, the proposed method could be carried out without changing N .

3) EVALUATION WITH TWO-ILLUMINANT DATASET
In Sections IV-B, IV-C1, and IV-C2, we prepared five images taken under various illumination conditions. However, the images were far different from those taken under general scenes. To test the proposed method under usual photographing conditions, we used the two-illuminant dataset [43] and compared the proposed method with single-illuminant WB [58] and multi-illuminant WB [25], [42], [43], [45], [48], [49], [51]. Also, the proposed method, single-illuminant WB [58], and methods for multi-illuminant scenes [42], [43] were carried out with nine illuminant estimation algorithms: White-Patch (WP) [11], Grey-World (GW) [12], Shadesof-Gray (SoG) [15], 1st-order Grey-Edge (GE1) [19], 2ndorder Grey-Edge (GE2) [19], PCA [24], Grey-Pixels for single-illuminant estimation [GP(std)] [25], Mean-Shifted Grey-Pixels (MSGP) [31], and Quasi-Unsupervised Color Constancy (QUCC) [32]. Table 3 shows the angular errors for images in the two-illuminant dataset. From the table, the proposed method outperformed single-illuminant WB, and it had almost the same performance as multi-illuminant WB. In particular, the proposed method had a lower angular error than Gijsenij et al.'s method [42], MIRF [43], Grey-Pixels   for multiple illuminants [25], Zhang et al.'s method [48], and BU-MCC [49]. For the laboratory set, Hussain et al.'s method [45] had a slightly lower error than the proposed method because the optimal number of segments (= 4) was prepared for this experiment. However, if a different number of segments were chosen, the performance of Hussain et al.'s method would decrease [45]. In contrast, the proposed method can maintain a high accuracy under various N values as described in Figs. 7 and 8. For the real set, TD-MCC [49] and Bianco et al.'s method with a neural network [51] also obtained competitive performance compared to the proposed method because a large amount of carefully prepared training data could be used.
In addition to the angular error, adjusted images should be subjectively compared. Figure 9 shows adjusted images for four sample images from the dataset. When we subjectively compared the proposed method with the conventional multi-illuminant WB [42], [45], the proposed method showed a more improved color constancy than the conventional methods despite some of the higher angular errors. Therefore, we confirmed that the proposed method is also effective for images where general scenes are captured.

4) EVALUATION WITH STATE-OF-THE-ART DATASETS
In this experiment, we evaluated the performance of the proposed method by using the LSMI dataset [65] and Afifi et al.'s mixed-illuminant test set [46]. In this experiment, the method was compared with single-illuminant WB [58], multi-illuminant WB [25], [42], [45]. Also   The proposed method and single-illuminant WB [58] were carried out with five illuminant estimation algorithms: White-Patch (WP) [11], Grey-World (GW) [12], 1st-order Grey-Edge (GE1) [19], PCA [24], and Grey-Pixels for single-illuminant estimation [GP(std)] [25]. Tables 4 and 5 show experimental results on the LSMI dataset and Afifi et al.'s mixed-illuminant test set, respectively. From the tables, the proposed method outperformed the conventional methods. Therefore, the effectiveness of the proposed method was confirmed under the use of the state-of-the-art datasets.

V. CONCLUSION
In this paper, we proposed a novel white balance adjustment for single and multiple illuminants including non-uniform light sources. The proposed method, called ''N-white balancing,'' maps N source white points into a ground truth one. While traditional single-illuminant white balancing considers adjusting a source white point under a light source, N-white balancing can reduce lighting effects under multiple illuminants in addition to single ones. Only a handful of methods have been proposed to correct multi-illuminant scenes; however, in several methods, the exact number of light sources or the exact number of scene segments need to be known in advance. In contrast, in the proposed method, N does not need to correspond to the exact number of light sources or segments. Additionally, color distortion is never caused in the proposed method because a diagonal matrix is used, unlike multi-color balancing. In experiments, the proposed method and the conventional methods were evaluated by using various illumination conditions. The results show that N-white balancing outperformed the conventional methods. By improving the accuracy of color correction under various illumination conditions, the proposed method is expected to contribute to increasing the number of color-based applications in the future. In future work, we would like to demonstrate object recognition and other applications by purposely using the color information adjusted by the proposed method.