A Multiscale Objective Function for Camera Color Correction

Color correction (CC) plays a pivotal role in camera imaging. Existing approaches usually conduct CC tuning by minimizing ∆E (e.g. ∆E2000), a standard metric proposed by CIE for representing color differences in LAB space. However, we observe that not all the colors with identical ∆E error to the target color have with same perceptual preference. Consequently, optimizing CC by minimizing ∆E solely does not always produce satisfactory color-rendition accuracy. To deal with the problem, in this paper, we propose a new score function, namely Ψ, for a more accurate discrimination of different color-rendition mappings. This is achieved by a multi-scale objective incorporating not only ∆E, but also ∆H and ∆C, which respectively indicate color differences from hue and chroma perspectives. We describe the details of Ψ and show how to adjust its parameters for different preferences. We verify the usefulness of Ψ in experiments by embedding it in various CC tuning algorithms. The empirical results show that Ψ consistently leads to better color-rendition accuracy not only in training but also in validation sets. Finally, we deploy our new objective for tuning a real-world commercial digital camera and show that it delivers improved performance.


INTRODUCTION
Color correction is an essential component in digital cameras because of the fact that camera sensor sensitivities cannot be represented as the linear combination of CIE color matching functions [1].It is known that this limitation causes camera-eye metamerism [2], i.e., under certain illumination, some surfaces with different colors to the eye might induce the same camera responses.While color correction cannot resolve metamerism, it helps alleviate the issue by generating the best mapping from camera RGBs to device-independent XYZs or display sRGBs [3].
Despite the fact that standard metrics like ∆E can provide color measurement generally consistent with human perception, we observe that in terms of guiding optimization, they are still far from ideal.Assume that a target color is noted as q * , and a corrected color is q, it is true that ∆E(q * , q) = 0 implies that q is a perfect rendition of q * .However, imagine that there is another color prediction q, and ∆E(q * , q) = ∆E(q * , q) > 0, does that mean q and q are equally good?Obviously, in many cases the answer is negative.It is then a question of how should we compare the relative quality of q and q.Naturally, we might resort to other measurements other than ∆E for further assessment of q and q.In this paper, we use chroma difference ∆C and hue difference ∆H, which we assume that they can characterize color differences from chroma and hue, respectively.Like ∆E, a smaller numerical value represents a smaller color difference for ∆H, while ∆C is a ratio measurement that is preferred to be close to or slightly above 1.0.Figure 1 shows three pairs of examples on how these two metrics can help identify better color-rendition.
However, incorporating two new measurements for color assessment brings a new challenge to optimization.A straightforward solution is to employ multi-objective optimization, the drawback is that those measurements by ∆E, ∆H, and ∆C are generally not equally important.Ideally, as Figure 1 shows, we prefer to refer to ∆H and ∆C only when ∆E is insufficient to provide reliable ranking for cor-Fig.1. Examples showing the deficiency of ∆E 00 .For each pair, the left one has either an equal or even larger ∆E error than its right counterpart, but visually the left one is better.rected colors.For this aim, we propose a multiscale score function dubbed Ψ.That is, it uses ∆E, ∆C, and ∆H at different scales.The rationale is, for example, when ∆E is very high, there is no need to use ∆C and ∆H at all, since all predicted colors at this range shall be regarded as terrible.By this scheme, Ψ effectively partitions the predicted colors into many different zones, and at each zone, it refers to different measurements for comparing color qualities.In practice, achieving ideal color correction could be difficult, it is desirable for digital cameras to maintain a consistent correction style.To support this, we introduce hyper-parameters to Ψ for practitioners to swiftly switch between different preferences.To demonstrate the effectiveness of our score function Ψ, we implement it with several popular CC tuning algorithms [20,24], and compare its performance with the best existing tuning objective ∆E 00 (see [20]).Experiment results show that our proposed Ψ score function guides the tuning more effectively, producing more visually accurate color correction results.

PROBLEM STATEMENT
Formally, color correction via linear mapping can be generally formulated as where p ∈ R N is the input vector formed by camera RGBs, q ∈ R 3 is the output vector in the target color space, and M ∈ R 3×N is the color correction matrix.LCC directly uses camera RGB values as the input where N = 3. PCC [15] extends the input with polynomial terms, where N = 9 and 19 for the second and third-order PCC.Meanwhile, RPCC [6] proposes root polynomial terms in the input vector, which gives N = 6 and 13 terms for second and third-order RPCC.Since RPCC has been proven to significantly reduce colorimetric error while being exposure-independent, we adopt RPCC as the formulation in this paper.
The matrix M ∈ R 3×N is typically obtained by optimizing its 3 × N elements using an objective function E(•) as where Q * ∈ R 3×K represents target color set of size K, and P ∈ R N ×K is K training examples and each with feature size of N .A number of choices can be considered for E.
Traditional approaches [6][7][8]17] usually use XYZ values as the target and minimize squared error or angular differences to calculate M .Following work [13,18,19] show that ∆E defined in the LAB color space, especially ∆E 00 , gives better results as it takes into account the human eye's perceptual sensitivity at different colors.To simplify notation, in this paper, we refer to ∆E as a shorthand of ∆E 00 .

NEW OBJECTIVE FOR COLOR CORRECTION
Given a color (a, b, l) represented in CIE LAB color space, we employ its polar coordinate representation, denoted as CIE HLC, where the hue h and chroma c are defined as Then we have the relative measurement in chroma and hue defined as , where c and ĉ are the target and predicted chroma,

Multiscale Score by Ψ
First, we design Ψ ∈ [0, 1], where 0 and 1 respectively means worst and perfect scores.We introduce two highlevel parameters ∆E max and ∆E min , which simply partitions the evaluation into three parts: 1) those greater than ∆E max , 2) those between ∆E min and ∆E max , and 3) those below ∆E min , which we may refer each of them as R 1 , R 2 and R 3 , respectively in Figure 2. The potential choice of ∆E min could be a just noticeable threshold, and ∆E max can be the threshold for unacceptable difference value.E.g., {∆E min , ∆E max } = {2, 6}.Clearly, we expect scores to follow To achieve this, we use an upper-and lower-bound controlled function F U L , e.g., the upper bound for R 1 becomes the lower bound for R 2 .We set those bounds as {0, 0.4, 0.9, 1.0} such that scores for R 1 , R 2 and R 3 respectively fall into [0, 0.4], [0.4,0.9] and [0.9, 1.0].The full definition of Ψ is as follows.
As shown in the contours in Figure 2, R 1 stands for those outside of the larger ellipse, R 3 is the inner ellipse and R 2 is the region between outer and inner ellipses.Most critical region is R 2 where we further partition it into sub-regions R 2.1 , R 2.2 , R 2.3 and R 2.4 by considering the product of two binary conditions whether ∆H > ∆H max and whether ∆C > 1.For those points inside of the same region, we rely on a standard exponential decay function F for scoring, defined as below.
In LAB space, rotating the horizontal line results in color changes.Therefore, given X-rite 18 color card, it corresponds to 18 straight lines crossing the origin and partitioning the space into 19 angular zones.Clearly, for a target color q * , and two candidates q 1 and q 2 , we would prefer q 1 than q 2 if q 1 and q * are in the same angular zone while q 2 is not.Taking this intuition into account, we use binary variable z to represent this information.The function g is a look-up table shown in Table 1.The function F is an exponential decay function parameterized majorly by α, which we set differently for each region.The predictions with lower hue errors shall be more preferred, we thus assign these with higher scores via a larger decay constant α.That is, α 2 , α 3 should be larger than α 4 , α 5 .
Then, the predictions with lower chroma errors shall receive higher scores, which can be achieved by setting α 2 ≥ α 3 and α 4 ≥ α 5 .In summary, Ψ can be viewed as a generalization of ∆E: in the special case, ∆E min , ∆E max are 0, 100 respectively, and all α i parameters are close to 0, then Ψ becomes a linear normalization of ∆E to [0.0, 1.0].

Hyper-parameters
The score function Ψ has two groups of parameters.The first group is {∆E min , ∆E max , ∆H max }, which coarsely decides how strictly we wish to tune the CC pipeline.Those parameters can be easily set by the users with regard to their own tuning platform.The second group is fine-level parameter, including all the α values and the angular partition of LAB space with color patches.We recommend the following setting shown in Table 1.Higher α corresponds to a slower decay and α = 0 means a linear decay.Adjustment of these values reflects fine-level tuning on the expectation of results.For example, in commercial cameras, other than smaller rendition error on average, color tone consistency might be another important factor for real-world user experience.Ψ offers the opportunity for casting this requirement by modifying its parameters.

EXPERIMENTS
To validate the efficacy of Ψ, we perform both synthetic and real camera-in-the-loop color correction experiments.

Performance on Tuning Synthetic Pipeline
For synthetic case, we use DXC-930 Camera [26] sensor sensitivities to integrate the spectral data from 140 patch X-rite SG color chart (81 patches remain after excluding gray patches), and surfaces collected from SFU dataset [26].
Figure 3 shows the overall workflow, where the transformations we used are from [27] for calculating camera sensor response and ground truth values under D65 illuminant.To fully show the advantage of Ψ over CIE ∆E as CC tuning objective, we select 5 popular tuning algorithms including simulated annealing, hill climbing, Powell's method, particle swarm optimization, and evolutionary search.For all cases, as in [6]  ∆E and Ψ separately and compare their performance.The tuning is conducted on X-rite 140 color checker card, while the test is performed on SFU dataset.We refer to two metrics for assessing the generated CCM's quality, i.e., ∆E and ∆H.We report both averaged value and maximum value of ∆E and ∆H.As in Table 2, for training, in terms of ∆E, only in the PM case Ψ leads to a larger mean ∆E error; for all other cases, Ψ has either equal or smaller mean ∆E.For testing on SFU dataset, only with simulated annealing, Ψ has larger ∆E error, presumably because that, SA is too simple such that it is unable to effectively search for promising results guided by Ψ.For strong optimization algorithms such as Powell's method and Evolutionary search, the tuning qualities using either ∆E or Ψ are superior to other algorithms.This observation is consistent with previous research that high-quality optimization algorithm is also an important component for CC tuning.For both PM and ES, we see that results by Ψ are all better than using ∆E.The most significant improvement for Ψ lies on max ∆H, which we see that those of Ψ are all > 40% smaller than those of ∆E.This indicates that Ψ not only leads to smaller averaged error, but also tends to guide the search to produce more robust correction.This is desirable for real-world cameras, since enhancing worst-case performance also improves user's experience.

Camera-in-the-Loop Tuning
For real-camera-in-the-loop scenario, we deploy Ψ with evolutionary search with 2500 search budget for tuning a Huawei multimedia edge device.As before, we use 140 X-rite SG color checker as the tuning scenario.Using Ψ led to improvement over the default CC matrix by 16.3% in averaged ∆E and 29.4% in averaged ∆H.As in Figures 4 and 5, our method produces better color-rendition than iPhone 13 Pro-Max with smaller averaged ∆E and ∆H. Figure 4 shows some example patches where the recovered colors are visually closer to ground truth.

CONCLUSION
We have presented a new objective for better color correction tuning.Different from standard metrics, such as ∆E (e.g., ∆E 76 , ∆E 94 , ∆E 00 ), which are proposed as general evaluation for assessing color differences, our new score function is designed exclusively for the aim of reliable comparison of different color candidates in the presence of a ground-truth.We expect that the new score function would be more suited for color correction tuning, and experiment results support our hypothesis.It is important to note that, however, Ψ is not introduced to replace metrics such as ∆E, instead, it should be more appropriately viewed as a complement of ∆E, particularly for tuning color correction pipelines.In our study, it helps generate not only more accurate color-rendition accuracy but also leads to more balanced results.While there are other approaches trying to produce more robust color correction [21], these methods are orthogonal to our work.So far, we have only focused on tuning color correction matrices.In reality, color correction is heavily affected by other modules such as white-balancing [28].In the future, we aim to study and improve color correction while considering these processing modules.

Fig. 2 .
Fig. 2. Illustration of Ψ function.Innermost point is groundtruth.Ψ further partitions R 2 into four sub-regions.Due to information by ∆C and ∆H, the preference ranking of regionsin R 2 is R 2.1 > R 2.2 > R 2.3 > R 2.4 -lighter color (closer to white) indicates higher score according to Eq. (3).

Fig. 3 .
Fig. 3. Illustration of X-rite SG color checker and synthetic pipeline for color correction.

Fig. 5 .
Fig. 5. Visualization of X-rite results in LAB space for camera-in-the-loop experiments.