Development of Bayesian Neural Network Model to Predict the Correlated Colour Temperature Using Digital Camera and Macbeth ColorChecker Chart

Information about correlated color temperature influencing the scene due to the surrounding lighting is vital, especially for circadian lighting and photography. This paper proposes a novel image-based machine learning model to predict the correlated color temperature in a scene with the help of the Macbeth ColorChecker color rendition chart and a DSLR camera. In the proposed technique, the researcher fixes the white balance setting in the camera, thereby forcing color difference in the captured image of the Macbeth ColorChecker chart placed in the scene. The Bayesian neural network model considers the color difference values of the six spectrally neutral patches of the Macbeth ColorChecker chart as inputs for CCT prediction. The color differences are calculated using the CIEDE2000 color difference formula. Fours models with white balance settings in the camera as 5000 K, 6500 K, 8000 K, and 10000 K were developed and analyzed. It is experimentally found that the correlated color temperature prediction error is less than five percent by the proposed model with white balance setting in the DSLR camera as 10000 K. The proposed model performed consistently during varied lighting levels and mixed CCT lighting conditions set up with LED, incandescent, and fluorescent lamps.


I. INTRODUCTION
Correlated color temperature (CCT) expressed in Kelvin (K) indicates the color of the influencing light in a scene. The knowledge of the CCT value can be used to estimate the appearance of a known object in a scene. In various sectors like healthcare, industry, and corporate, circadian lighting is gaining importance, which demands color tuning of indoor artificial light. [1]- [3] This is achieved by varying CCT and intensity of the artificial light in sync with outdoor light conditions. In photography, knowledge of light source CCT is crucial in determining the color output of the photograph. The camera setting 'white balance', which can be set by the photographer, is associated with the CCT of light governing the scene. [4] Hence, CCT information of the light source in a location is vital in human-centric lighting and photography applications.
The use of digital cameras as sensors for photometry has gained momentum and importance due to their costeffectiveness, measurement speed, and accuracy. [5]- [7] Techniques of using digital single-lens reflex (DSLR) camera for luminance measurement are well explored and established. [8]- [10] This motivates to explore further the possibility of using a digital camera to estimate photometric parameters like CCT, CRI which are helpful to access the qualitative aspects of lighting design.
The traditional approach of CCT measurement involves placing the artificial light source in an integrating sphere that houses a spectrometer. Handheld spectrometers are now available in the market with which CCT of the influencing light source on the work plane can be obtained. But both methods require expensive calibrated instruments. Professional photographers use standardized target surfaces to ensure color reproduction in the resulting photograph.
CIEDE2000 has been officially adopted as the color difference formula by CIE. [11], [12] Software tools are available to compute this color difference caused by the camera settings during image capture. The approach of using standard white balance cards and ColorChecker charts for color rectification is enhanced with the help of postprocessing photo editing software commercially available. [13] Though these charts help correct color in the resulting image, they do not provide the actual numerical value of CCT in the captured scene.
When a camera is used for photometric measurement, the white balance setting plays a significant role in deciding the pixel values of the resulting image. The user must carefully analyze the lighting condition in the scene and set the white balance mode. Keeping auto white balance (AWB) mode on the camera may lead to unreliable pixel data for photometry. The post-processing image manipulation algorithms of the camera also contribute to the pixel values. It must be noted that the user has limited control over these image manipulation steps. A sensor calibration method is suggested by Juan Sebastian Botero et al. [14] to address the above issue. Sneha Jain et al. [15] photographically obtained CIE XYZ values using HDR images. McCamy's method is used to calculate CCT values from CIE XYZ values. The percentage error reported by the authors for CCT prediction is ±9.54%. The light source in the scene was a tubular daylight device. The radiance daylight simulation tool was used to get X, Y, Z pixel values from the HDR image.
Authors of literature [16] proposed a novel model that can predict dynamic variations of CRI and CCT due to system power variations of phosphor-coated white LED systems. Electrical, optical, spectral, thermal, device-specific parameters were extracted from steady-state tests only. The authors claimed that the model could be used during steadystate and dynamic situations. The current article proposes a machine learning-based approach to predict the actual CCT influencing the scene with the help of Xrite ColorChecker chart pixel data.

II. PROPOSED HYPOTHESIS
This study explores the possibility of establishing a relation between the pixel data of the ColorChecker chart image and the CCT of light in the scene and proposes a machine learning-based model to predict the CCT influencing a given location.
The hypothesis statement is as follows: 'When the white balance of a digital camera is manually fixed, and an image of the ColorChecker chart is captured, the color difference of the spectrally neutral patches of the chart that result after shifting the pixel values to D50 white point can be used to predict the actual CCT of the surrounding light'.
Once the white balance value is fixed in the camera, say to 10000 K, the camera's image processor considers the surrounding light condition as cool white (with CCT 10000 K). The processor then compensates by adding a yellow tint to produce an image of the scene under neutral white conditions. For example, if the actual lighting condition of the scene is warm white, say 3200 K, and the image of the scene is captured with a white balance setting as 10000 K, a considerable yellow tint appears in the resulting image. The image seems warmer than seen by the eye. This condition is due to the additional yellow tint compensation made by the camera image processor to make the image scene appear neutral. The compensation done by the camera for neutralizing the image is assumed to be directly proportional to the difference between the actual CCT of the scene and the set white balance in the camera. This process is not desired by a photographer whose aim is usually to capture a scene as seen by the human eyes. In the proposed technique, the color difference caused by the white balance setting in the resulting image from the actual scene is desired for analysis purposes.
The CIE L*, a*, b* values of patches of x-rite ColorChecker chart classic manufactured before and after November 2014 are made available by the manufacturer. [17] These standard reference values are noted by illuminating the chart with a D50 illuminant, that is, standard lighting condition with a CCT value of 5003 K. The reference L*, a*, b* values of the six neutral patches used during experimentation is given in Table 1. To compute the color difference, it is necessary that reference and measured L*, a*, b* values should have the same white point. [18] Hence, during the experimentation, measured L*, a*, b* value under a lighting condition is transformed to L*, a*, b* value with D50 white point.
The experimental setup to test the proposed hypothesis, model development to relate color difference values with actual CCT, and results obtained are discussed in the following sections.

III. EXPERIMENTAL SETUP
The ColorChecker chart is exposed to a wide range of CCT values from warm white to cool white lighting conditions in the experimentation. For every lighting condition, the image of the ColorChecker chart is captured using a DSLR camera, and the actual CCT value is measured using calibrated Asensetek lighting passport spectrometer. The actual values of CCT measured from the standard meter and the extracted RGB values of the spectrally neutral targets in the ColorChecker chart are used to develop a supervised learning-based neural network model for predicting CCT. The workflow of the data building process is shown in Fig. 1.

A. SURROUNDING CONDITIONS
For gathering data for the proposed model, a dark room is used to ensure consistent lighting conditions. Experimenting in the darkroom nullifies the influence of wall surfaces and objects on the ColorChecker chart. Also, the ColorChecker chart is illuminated only by the desired sources for the research.
In the setup, four commercially available LED battens are used. The technical details of the fixture are shown in Table  2. With the help of the android app, the CCT of the four lamps can be changed in steps of 1% simultaneously, from 2800 K to 5800 K, providing 110 unique CCT lighting conditions. The LED fixture's spectral power distribution (SPD) when set to cool white and warm white conditions is shown in Fig. 2. The app also facilitated the fixture's remote operation, thereby avoiding the possible error due to light reflection from users' clothing. Care is also taken to provide a constant voltage supply to the lighting fixtures.  The light fixtures are vertically mounted to throw light on the ColorChecker chart positioned 1.5 meters away. The ColorChecker chart (passport) is clamped to cardboard positioned vertically. The calibrated spectrometer is mounted above the chart to capture the true CCT due to the light fixtures, as shown in Fig. 3. The spectrometer is also triggered to capture accurate data remotely through spectrum genius mobile application. The isometric view of the light fixture set up with the camera is as shown in Fig. 4.  The camera is positioned behind the lighting fixture to avoid glare on the camera lens. The image of the ColorChecker chart is captured by placing it in front of the lighting fixture.

B. CAMERA SETUP
The camera used for the experiment is a full-frame Nikon D750 with a 50 mm Nikkor F/1.8D AF prime lens. In addition to standard white balance settings like auto, cloudy, shade, the user can manually set the color temperature value of the scene in the camera. The motive behind using a prime lens is to reduce the possibility of vignetting and distortions due to the series of glasses found in telephoto lenses. Table 3 shows camera settings made during data collection. ISO 100 camera sensor sensitivity leads to less digital noise in the resulting image. The sensor is correctly exposed (exposure compensation is zero) using aperture priority mode. Since the chart surface is parallel to the image sensor, the aperture of f/3.2 ensures the correct focus of the ColorChecker chart patches in the image. Post-processing settings like 'active D-lighting' and 'picture control' are minimized. The camera is mounted on a tripod. The center of the lens and ColorChecker chart are in the same line. The camera flash is kept OFF, and the shutter button is remotely triggered using a wireless remote.

C. IMAGE DATA EXTRACTION AND MODEL DEVELOPMENT
The proposed model should work in any lighting conditions caused by different types of light sources (LED, incandescent, fluorescent). Four models are developed using images captured with white balance as 5000 K, 6500 K, 8000 K, and 10000 K. For each white balance setting, around 110 images are analyzed. The white balance value that gives the best performing model has to be fixed as the white balance setting during the deployment of the proposed setup. In sections D, E, and F, the procedure followed to extract data from images, the statistical significance analysis, and model development are discussed.

D. DATA EXTRACTION FROM IMAGES
Data extraction from images is a laborious process without proper image processing tools. The steps to obtain the CIEDE2000 color difference value for each of the neutral patches of the ColorChecker chart are shown in Fig. 5.

FIGURE 5. Stages to calculate the color difference of a neutral color patch
An application is developed using MATLAB App Designer to simplify data collection and tabulation. Fig. 6 shows the opening page of the application when the user chooses an image to analyze using the 'Browse Image' button. VOLUME XX, 2017 9

FIGURE 6. Opening page of the developed GUI
The application is used in the following steps: Step 1: Browse and select the image to be analyzed Once the user chooses the image to analyze using the 'Browse Image' button, camera aperture, shutter speed, ISO settings used to capture the scene, and the file name are displayed at the bottom of the application screen to facilitate the user. The EXIF data of the image file using image processing toolbox commands is used for this purpose. The user can also enter the true CCT and CRI values noted from the calibrated sensor during the capture of the scene.
Step 2: Color checker chart identification The user must manually specify the 'registration points' of the ColorChecker chart, as shown in Fig. 7. This is done by moving the four blue points (labeled as 1. Black point, 2. White point, 3. Brown point, and 4. Bluish-green point) to their marker position available in the chart. Once this is done, the 'Marker points updated' button must be pressed.

FIGURE 7. Specifying the ColorChecker chart registration points
On the press of the 'Marker points updated' button in the designed app, the following operations occur: 1) All the patches of the ColorChecker chart are identified, and RGB pixel values of the patches are measured. This step of chart recognition can be verified, as shown in Fig. 8.
2) Color differences based on the CIEDE2000 formula and CIE94 formula are computed and tabulated

FIGURE 8. Identification of ColorChecker chart
The above steps occur due to 'ColorChecker' (function of MATLAB available since 2020b version), 'measureColor' (measures RGB values from the chart patches), 'rgb2lab' (to covert RGB color space to LAB with D50 as white point), 'imcolordiff' (to calculate color difference with CIEDE2000 formula). Depending upon the manufactured date of the ColorChecker chart (before/after November 2014), the reference values used in the 'measureColor' function are to be modified.
Step 3: Data saving After step 2, the application user interface appears, as shown in Fig. 9.
To keep track of number of images analysed To export data to an excel file

FIGURE 9. User interface after the chart is analyzed
'Export Data' button is enabled to export the following data to an excel file: The excel file can be accessed from the 'current folder' set in MATLAB. After the data is exported, the user can continue to analyze another image. A tracker is maintained to inform the user about the number of image files analyzed.

E. STATISTICAL ANALYSIS OF THE DATA
The data of color difference measured for six neutral patches and actual CCT condition of the scene is recorded and tabulated for each white balance setting on the camera (5000 K, 6500 K, 8000 K, and 10000 K). Statistical tests are carried out to check if a statistical correlation between the tabulated color difference values of the neutral patches and the actual CCT value of the scene exists. Table 4 shows the outcome of Pearson's correlation test for the color difference vs. actual CCT data collected corresponding to a white balance setting of 10000 K. A negative 'r' value close to 1 indicates good inverse relation between the two variables supporting the possibility of a mathematical model for predicting the CCT of the scene.
In addition to the correlation test, z-score values were used to detect outliers. Measurement with z-score above and below 2.56 and -2.56, respectively, are considered outliers.

F. TRAINING A BAYESIAN NEURAL NETWORK
The CIE2000 color difference data of the six neutral patches populated is used for training the Bayesian neural network. Four neural network models corresponding to 5000 K, 6500 K, 8000 K, and 10000 K white balance settings are developed. The network structure is as shown in Fig. 10. 75% of the data is used for training, and the remaining 25% is used for testing purposes.

IV. RESULTS AND DISCUSSION
This section discusses the results obtained from the Bayesian neural network-based model for four different manual white balance settings in the camera. Fig. 11-14 shows the error histograms, training, and testing regression plots for four different white balance settings in the camera, i.e., 5000 K, 6500 K, 8000 K, and 10000 K. Table 5 provides the Mean Square Error (MSE) of the CCT prediction models during training and testing. When the white balance is set to 5000 K, it is observed that the model is overfitting for the training data hence leading to poor generalization. At 6500 K white balance setting in the camera, the performance improves. But for cases with actual CCT values close to 6500 K, the model prediction errors lead to poor testing MSE.
It must be noted that most of the artificial light sources operate in the 2500 K to 6500 K range. The model's prediction performance with a white balance setting in the camera at 8000 K and 10000 K is very encouraging.
The model's mean square error (MSE) with a white balance of 8000 K is better than 10000 K. But prediction instances near-zero error line is more from the model developed with 10000 K white balance. Also, MSE from the 10000 K white balance model testing samples suggests superior performance than the other model in question. The spectral power distribution of traditional light sources like incandescent bulbs and fluorescent lamps significantly differs from LED light sources. The model has a scope of bias since it was developed from an LED smart batten measurement data. This problem is addressed with additional light measurements using traditional sources and a different LED fixture in the darkroom, whose SPDs are shown in Fig.  15. The 10000 K white balance-based model is re-trained with these additional measurements. These added measurements ensured a versatile model for CCT prediction. The robustness of the model is evaluated in two ways. One, by creating lighting scenes with LED lamps operating at different CCT values in a given scene. The model's performance is tabulated in Table 6. Two, measurements were carried out under different light sources and light levels. The results are shown in Table 7. The ColorChecker chart images were taken with the camera's white balance setting set to 10000 K. The proposed model gave steady performance even during dimming conditions, as shown in Table 7.

V. CONCLUSION
A novel technique to predict the correlated color temperature in a scene due to the lighting condition of the surrounding using the camera as the sole sensor is proposed in this paper. Macbeth ColorChecker chart is placed in the scene, and the image is captured using a DSLR camera. The pixel data of the six spectrally neutral patches of the ColorChecker chart is used to compute color difference based on the CIEDE2000 formula. The bayesian neural network-based model predicts the CCT value using the color difference values of six neutral patches as inputs.
The authors would like to emphasize that the white balance setting in the camera plays a crucial role when the image is used for photometric analysis. The resulting pixel values of the image due to incorrect white balance set automatically or manually will lead to erroneous measurements. For the proposed model, it is experimentally found that color difference values obtained with white balance setting in the camera as 10000 K provided the best fit than that of 5000 K, 6500 K, and 8000 K. The correlated color temperature prediction error is less than 5%, with a white balance setting of 10000 K. The prediction error was within the limit even when the scene was illuminated with different CCT values sources. The model's performance is consistent for a wide range of illuminance levels. A DSLR camera with a fixed focal length lens (prime lens) is employed during the study. The authors believe that the proposed technique can be carried out using calibrated mobile phone cameras, thereby reducing the overall cost of the system. The accuracy of the model can be further improved with more measurements and re-training. Reducing the complexity of the model by decreasing the number of inputs to the neural network can be explored. The RAW image information might further enhance data quality.