Extraction and Classification of Surface Color Information Using Selective LED Illumination and a Statistical Neural Network

The color of a product is one of the main factors that affect consumer choices. However, it can be difficult to discern the subtle differences in color in order to inspect or classify products without expensive and specialized equipment. In this study, we proposed an approach where standard off-the-shelf components (digital camera and RGB LED lights) can be used to robustly classify products based on their color. By varying the color of the illumination, environment-independent information of the color of the object can be obtained. Then, using a Log-Linearized Gaussian Mixture Neural Network (LLGMN), a neural network based on a statistical model, the objects can be effectively and robustly classified based on their color without requiring a large number of training data. An experiment was performed to verify the effectiveness of this approach. 8 samples with only a subtle difference in color were prepared, and a prototype was developed to be used to classify these 8 samples. We demonstrate that we are able to classify the 8 samples an accuracy of 100% when there was no ambient light and were still able to maintain an accuracy of 94.1% when there was ambient light in the room.

SECTION I. Introduction

The color appearance of a product can have a significant effect on consumer choices. Therefore, producers have strong incentives to maintain consistency in the surface color of their products. In the case of manufactured goods, there is a need to inspect their products to ensure the colors are even throughout and that there is minimal variation between batches [1], [2]. For natural products that are mined or harvested, such as ceramic tiles [3], wood panels [4], [5], producers often need to classify and sort them based on their color. Furthermore, color is one of the main attributes used by consumers when evaluating the safety and quality of food products [6], [7], [8], [9].

Traditionally, this inspection and classification was done by humans using only their naked eyes. In addition to the high labour costs and relatively low speeds of this approach, there is also often variability between human observers [10], [11]. This variability can increase substantially when different light sources are used, even when the color of these light sources appears the same. These metameric light sources can have very different spectral distributions [12], leading to radically different colors being perceived [13], [14], [15]. This is especially a problem in recent years, where LED-based lighting is becoming increasingly prevalent due to their cost and energy advantages. The white light in these lighting systems is produced from a combination of narrowband or spikey spectra sources [12], [16], which can change the color appearance of objects. There is a need for an automatic visual inspection system that is robust to variations in ambient light and can discern subtle color difference.

We propose a system utilising of off-the-shelf components, specifically, RGB LED lights and a digital camera, in combination with a Log-Linearized Gaussian Mixture Neural Network (LLGMN) [17] to automatically classify the different grades or classes of a product.

This study is structured as follows. First, Section II discusses the works related to our research the advantages of our approach. Section III introduces the proposed method’s system components and the LLGMN algorithm encompassing the statistical structure. In Section IV, verification experiments will be conducted using plastic parts colored with blue paint containing slight color differences. In the experiment, we verify the effectiveness of LLGMN to identify the color information array extracted by the proposed method. Finally, Section VI presents a summary of this study.

SECTION II. Related Works

One often used solution to these metameric issues is through the use of conventional high-performance spectrometers to obtain the reflectance of these surfaces [18]. Although able to provide objective and accurate information about the surface color of an object under various lighting conditions, these devices are often expensive, have narrow field-of-view, and require a lot of space. Traditional scanning methods such as whiskbroom or pushbroom scanning can suffer from low speed due to their scanning mechanism, and snapshot spectral imaging methods are often bulky and expensive due to their complex optical systems [18], [19]. Additionally, they can also suffer from reduced pixel density due to spatial binning [20].

These characteristics of traditional spectrometers often make them unsuitable for many colorimetric applications. In today’s increasingly high-mix, low-volume manufacturing environments, where a large variety of products are produced in small quantities [21], there is a desire for flexible, low-cost, and portable systems that can be quickly repurpose and adapted to new products. Also, in applications where there are substantial benefits to the inspection or classification being performed in the field, such as when classifying the color of soil [22], [23], simple, portable, and robust devices are often preferred.

With technological advances in consumer electronics, digital cameras and RGB LED lights, where the intensity of its RGB channels can be individually controlled, are becoming ever cheaper and ubiquitous. Therefore, in this study, we propose combining these ubiquitous off-the-shelf components to produce a system that can objectively inspect and classify the colors of products. Digital cameras, on their own, with only red, green, and blue channel filters are often not sufficient to produce environment-independent measurements of surface color [24], [25]. Therefore, we propose making use of the narrowband RGB LED lights at different intensities to measure how the surface color changes. By the simultaneous multiplexing of the illumination and the imaging system using these off-the-shelf components, the environment-independent colorimetric properties of the surface can be determined, and products can be inspected for defect or to be classified into different grades.

Due to advances in the miniaturized, cost-effectiveness and efficiencies in digital camera and LEDs, the proposed system can be made portable and affordable, allowing them to be used flexibly and in the field. Additionally, the system can leverage advances in computer vision due to its use of a standard digital camera. For example, objects in raw image often need to be segmented before color inspection or classification; rice seeds need to be separated from its background before color identification [20], and the colors of early- and latewood of wood panels needs to be identified separately [4]. Furthermore, other visual inspection procedures, such as identifying defects, can be easily added to the same setup [26], [27], [28]. Although there has been research into using RGB LEDs with a digital color camera, their goal has been to reproduce a multi-spectral imaging system that can be used in placed of an imaging spectrometer [29], [30].

Finally, the classification approach proposed, LLGMN, does not require as many training samples as in general deep learning and has the excellent feature of building models based on statistical criteria, even from a few samples [31].

SECTION III. Methods

A. System Components

The proposed measurement system component is shown in Fig. 1. During the measuring process, the object is illuminated with color light (MD2-100UPRLGB, SHIMATEC Y.K.), and images are acquired using an RGB color camera (HD Pro Webcam C920, Logicool Co Ltd.). A computer (HP ProBook 430 G5, HP Development Co Ltd.) is connected to an RGB color camera and a microcontroller (Arduino Uno, Arduino, LLC Co Ltd.). The microcontroller is connected to a color light and a color light controller. The RGB color camera has a focal length of 3.67 [mm], an optical resolution of 3 [megapixel], and a maximum frame rate of 30 [fps]. The controller of the color light can control the intensity of each of the three LEDs (Red, Green, and Blue) in the color light by PWM control. The color light spectrum is shown in Fig. 2.

FIGURE 1.

System components.

FIGURE 2.

Color light specification.

B. Extraction of Color Information Array

In image acquisition, the brightness values in the 3 color planes (Red, Green, Blue) of the RGB color camera are measured when the object is illuminated in 7 different color light conditions to extract a multi-dimensional color information array of the object. The measurement algorithm is shown in Fig. 3.

FIGURE 3.

Color information array extraction algorithms.

The computer sends commands to the microcontroller via serial communication to control the intensity of the Red, Green, and Blue LEDs in the color light. As shown in Fig. 2, we use 3 monochromatic lights (Red, Green, Blue) and 4 colors (Yellow, Cyan, Magenta, White), which are a combination of these colors, for a total of 7 colors. The color light is switched every 0.4 [sec] and the image is acquired at 2.8 [sec] per object.

Each time the color light is switched, the brightness values in the Red, Green, and Blue color planes are measured. Brightness values are expressed in 8-bit(256) gradations. The image is acquired at $1280 \times 720$ [pixel]. Calculate the average brightness value of the object’s center in the image extracted by 100[pixel]$\times 100$ [pixel]. In other words, the 21-dimensional average brightness values of 7-color $\times$ RGB planes (3-planes) are extracted as a color information array for 1-object.

C. Discrimination Using a Statistical Neural Network

1) Log-Linearized Gaussian Mixture Neural Network

Color information arrays are identified using LLGMN, which is a neural network with statistical structures [17].

The structure of LLGMN is shown in Fig. 4. First, the feature vector $X\in {R}^{d}$ is pre-processed and transformed into the input vector $X\in {R}^{d}$ . The first layer consists of $H$ units corresponding to the dimensionality $H$ of the input vector $X$ and uses identity functions for the input-output functions of the units. The input-output relationship in the first layer is given by Equations (1) and (2), where $^{(1)}I_{j}$ is the input and $^{(1)}X_{j}$ is the output.

$$\begin{align*} Y_{k,m}& =\sum _{h=1}^{H}{^{(1)}}O_{h} w^{k,m}_{h} \tag {1}\\ ^{(2)}O_{k,m} & = \frac {\exp [Y_{k,m}] } {\sum _{\acute {k}=1}^{K} \sum _{\acute {m}=1}^{M_{K}} \exp [Y_{\acute {k},\acute {m}}]} \tag {2}\end{align*}$$ Note that $w^{K,M_{K}}=0$ .

FIGURE 4.

Structure of LLGMN.

The third layer consists of $K$ event units and outputs the posterior probability of event $k(k=1, \cdotp \cdotp , K)$ . Unit $k$ is coupled with $M_{k}$ units ${k,m}(m=1, \cdotp \cdotp , M_{K})$ in the second layer. The input-output relationship is represented by Equations (3) and (4).

$$\begin{align*} ^{(3)}I_{k} & = \sum _{m=1}^{M_{K}}{^{(2)}}O_{k,m} \tag {3}\\ O_{k} & = ^{(3)}I_{k} \tag {4}\end{align*}$$

The LLGMN is trained using $N$ sample data $x^{(n)}(n=1, \cdotp \cdotp , N)$ . Given $N$ sample data points (training data), the log-likelihood function $L$ is given by Equation (5).

$$\begin{equation*} L = \sum _{n=1}^{N} \sum _{k=1}^{K} T_{k} ^{(n)} \log ^{(3)} O_{k} \tag {5}\end{equation*}$$

The output value of the network $^{(}3)O_{k}$ corresponds to the posterior probability $P(k \mid x^{(n)})$ . For evaluation function $J$ , we use Equation (6), which is Equation (5) with a negative sign, and learn to minimize it, that is, to maximize the likelihood.

$$\begin{equation*} J = \sum _{n=1}^{N} J_{n} = - \sum _{n=1}^{N} \sum _{k=1}^{K} T_{k} ^{(n)} \log ^{(3)} O_{k} \tag {6}\end{equation*}$$

SECTION IV. Experiments and Results

Validation experiments were conducted to demonstrate the effectiveness of the proposed method. The following sections describe the experimental conditions, the improvement in identification performance with a multidimensional color information array, and the improvement in robustness of identification performance to changes in lighting conditions.

A. Experimental Conditions

The object is a plastic bottle cap painted with a color spray of a similar color. The color sprays are made by Tamiya Co Ltd. and are all blue in color. There were 8 variations, ranging from combinations that can be visually identified to those that are very difficult to identify. Blue color sprays are TS-10 FRENCH BLUE, TS-15 BLUE, TS-44 BRILLIANT BLUE, TS-50 MICA BLUE, TS-51 RACING BLUE, TS-53 DEEP METALLIC BLUE, TS-54 LIGHT METALLIC BLUE, and TS-57 BLUE VIOLET. As a preprocessing step, a primer was sprayed with a surfacer before spraying. FINE SURFACE PRIMER FOR PLASTIC & METAL (LIGHT GRAY) was used for the surfacer. The spectra generated on the object and the object are shown in Fig. 5. The number to the left of the object name in Fig. 5 represents the class number.

FIGURE 5.

Target objects.

The distance between the RGB color camera and the object was 90[mm]. The parameters for the RGB color camera were Brightness: 110, Contrast: 0.05, Exposure: 0.0088, Focus: 65, Gain: 0, White Balance: 4847, Saturation: 128, Sharpness: 25, Pan: 0, Tilt: 0, Zoom: 100. Sharpness: 25, Pan: 0, Tilt: 0, Zoom: 100.

B. Improved Identification Performance Using Multidimensional Color Information Array

Fig. 6 shows the color information that can be obtained using our proposed method. Images acquired under natural light of two target objects (6. TS-53 DEEP METALLIC BLUE and 5. TS-50 MICA BLUE) shown in Fig. 5 are shown on the left and the corresponding RGB intensity information under the seven color lights are shown to the right of each object. The brightness of each plane is converted to grayscale and displayed as a 256 grayscale image.

FIGURE 6.

Example of extraction color information using the proposed method.

The columns in each group represent the color light colors, and the rows represent the RGB planes. It was confirmed that the images in the upper and lower rows differ in appearance according to the color of the color light and the extracted planes. In particular, when the color light is Magenta, the images extracted from the Green and Blue planes were very different. The proposed method is expected to facilitate the identification of objects that are difficult to identify by visual inspection or traditional methods.

Next, the distribution of the average brightness values calculated from the extracted images, i.e., the color information array, is visualized in Fig. 7. Images were acquired for each of the eight target objects from 1 to 8. Images acquired under natural light are shown at the top of each graph. In each graph, 50 images are taken for the object and 50 points are plotted each. The color of each point corresponds to the color of the color light, the labels on the horizontal axis represent RGB3 color planes, and the vertical axis represents the average brightness value of each color plane. The multidimensional plots extracted by the proposed method are distributed with unique features for each object, indicating that each object can be easily identified from this information.

FIGURE 7.

Color band examples.

Furthermore, we compared the degree of separation of the extracted image information between the proposed method and common image feature extraction methods using F values. The proposed method lights the object with 7-color light and extracts a 21-dimensional color information array of $7 \times 3$ from 3 RGB planes. In contrast, the general method lights the object with white monochromatic light and extracts average brightness values of only 3 types from 3 RGB planes. Images were acquired 50 times for each condition, and the average brightness value was extracted. The comparison results are shown in Table 1. The F value of the color information array of the proposed method is larger. A large F value indicates a large variance in the average brightness value extracted among each object. Since a larger variance of the mean brightness value facilitates identification, we were able to show that the proposed method is superior to common methods in identifying the color of an object.

TABLE 1 Comparison results in terms of F-value.

C. Improved Robustness to Changes in Lighting Environment

In general, automated visual inspection requires a completely light-shielded environment to eliminate the effects of ambient light, but the equipment tends to be large. In actual manufacturing sites, it is often difficult to ensure a stable light-shielding environment due to sudden changes in manufacturing plans or production lines or to respond to inspections of irregularly manufactured products. It would be efficient if automatic visual inspection could be realized without needing a large light-shielding environment. The proposed method combines color light and color planes to extract a multidimensional color information array. It thus has the potential to achieve color identification that is robust to changes in ambient light, even in the absence of special light-shielding environments.

This study conducted experiments in two lighting patterns to verify whether the accuracy could be maintained when affected by ambient light. Images were acquired under normal living room conditions, with the fluorescent lighting in the room “on” and “off.” In both environments, images were acquired at night. The illuminance of the room was measured using an iOS application (QUAPIX Lite, Iwasaki Electric Co., Ltd.). The illuminance was 660[lx] with “room lights on” and 10[lx] with “room lights off.” The distance between the light and the object was 85[mm].

Table 2 and Fig. 8 show the data used in the verification experiment train the LLGMN model, and the experiment scene. The 4 conditions are shown based on 2 types of the lighting environment and 2 types of image feature extraction methods. As shown in Fig. 8, there are 2 types of lighting environments: “room lighting on” and “room lighting off”. There are two methods for extracting image features: the proposed method using a 21-dimensional color information array and the conventional method using 3-dimensional RGB information under white lighting. For each condition, 30 samples of training and test data were used.

TABLE 2 Condition of the experimental environment.

FIGURE 8.

Experiment scene.

We then attempted to classify the target objects under these four conditions using our proposed classification method. The parameters of LLGMN were set to 3 for the number of components, 8 for the number of classes, 30 for each class for the number of training samples, and 30 for each class for the number of test samples.

To compare the performance of our proposed method, we implemented two other classification methods. The first is a conventional method similar to LLGMN, Gaussian Mixture Model (GMM) [32], and the second is a convolutional neural network, ResNet101 [33].

GMM is a semiparametric model that allows flexible modeling from a smaller number of samples than deep learning. It is a method for modeling the statistical distribution of an object by mixing Gaussian distributions and is the basis of LLGMN. Because GMM is an unsupervised model, training and test data were mixed, and all 60 samples were used for modeling. The number of training sessions was set by trial and error to obtain the best test performance. The parameters of GMM were set to 3 for the number of components and 8 for the number of classes, as in LLGMN.

In recent years, ResNet101 is a popular convolutional neural network that is often used to classify images [34], [35]. The raw image under conditions 1 and 2 in Table 2 was used to train and test the accuracy of this method. To generate training and test samples for ResNet101, 2000 images were generated for each target object by randomly cropping $100 \times 100$ pixels around the center of the target object (within −50 to 50 pixels). A total of 16000 images were generated; 8000 images were used for training and the other 8000 images were used as test data. To input the images into ResNet101, these images were resized, and their intensities rescaled to $224 \times 224$ pixels and between 0 - 1. The weights of ResNet101 were initialised using the pre-trained ImageNet weights [36]. The model was trained with a batch size of 16, a stochastic gradient descent (SGD) optimizer and for an epoch of 20.

The experiment results are shown in Fig. 9. The graphs in columns 1 through 4 show conditions 1 through 4 of Table. 2, respectively, with the first row showing the results from GMM and the second row showing the results from LLGMN. Conditions 1 and 2 show that the proposed method using a 21-dimensional color information array achieves high identification accuracy. In particular, the LLGMN discriminates with 100% accuracy in both cases. However, when using the conventional GMM, the identification accuracy drops to 86.7% in Condition 2. This is presumably because the color information array did not allow for sufficient modeling because the samples’ distribution varied between the room’s “on” and “off” lighting conditions. Conditions 3 and 4 are the cases where no color information array is used, which is a common conventional method using only 3-dimensional RGB information under white monochromatic lighting. Under these conditions, the identification accuracy tended to decrease. In particular, the accuracy of the conventional GMM drops to 70.2% in Condition 3 and 50.4% in Condition 4. LLGMN maintains an accuracy of 94.1% in Condition 3 and 74.6% in Condition 4, even without color information array, due to its high modeling capability.

FIGURE 9.

Classification results of target objects.

ResNet101, on the other hand, achieved a classification accuracy of 99.2% and 90.5% for conditions 1 and 2. The proposed method, LLGMN in conditions 3 and 4 (using 7 color lights), achieved a 100% accuracy. Although the ResNet101 model achieved good results relative to other approaches when only white lighting was used, the proposed method of using 7 colored lights with LLGMN outperformed ResNet101.

SECTION V. Discussion

The experimental results demonstrated that the proposed color information array effectively improves the identification performance of slight color differences by expanding the color information to multiple dimensions compared to conventional general methods.

In addition, improved robustness to changes in the lighting environment was observed, confirming that LLGMN can be used to achieve stable and high accuracy. In other words, it has been proven that the system can stably demonstrate high identification accuracy without requiring a special light-shielding environment, even for data with only a slight change in color, which is conventionally difficult to distinguish.

As discussed in Sections I and II, illumination can have a substantial effect on the perception of colors. Hu et al. utilized deep learning to recognize the color of vehicles in natural scenes and found that most of the mistakes made were due to differences in illumination or indistinguishable colors [37]. Joze et al. proposed a solution to this color constancy problem by estimating illuminant colors and intensities by comparing the color of surfaces in the image to actual colors of known surfaces [38]. Our approach, although requiring additional hardware (RGB LED lights), does not make assumptions about the illumination, but instead imposes additional illuminations onto the object, increasing the information available to distinguish between colors. The increased information from our approach may enable classifications that are more robust external disturbances and be more sensitive to small differences in color.

The first limitation of this study is that artificial colored target objects were used. Although this simulated the challenges of discriminating between objects of similar colors in a manner which can be easily replicated, the colors used may not reflect that of actual products used in real-world industries. Nonetheless, we demonstrated that our approach of expanding the color information by using colored LED lights substantially increases the F-value of the target objects (ratio of the variance between the colors of the different objects to the variance of the observed colors of within each object).

Another limitations of this study is that the robustness of our method was only tested using ambient white indoor lights. The effects of different lighting environments and brightness, such as under direct sunlight or under warm lights, has not been investigated. Further research is needed to understand these effects and develop strategies to deal with these disturbances. However, the robustness of our method to indoor lights suggests that the use of a shade, which decreases the intensity of the surrounding ambient light, might be sufficient to maintain the classification accuracy of different colors using our proposed method.

SECTION VI. Conclusion

In this study, we proposed a system that uses a neural network LLGMN based on a statistical model to identify color differences by extending color information through image measurement using a combination of seven-color color light and RGB color cameras. This method can distinguish slight color differences, is robust to changes in ambient light, and is inexpensive and lightweight, making it easy to implement on existing lines. The experiment was conducted in two different environments, shaded and natural lighting, with eight samples having slight color differences that were difficult to distinguish with a typical RGB color camera. A comparison of the proposed and previous methods proved that the proposed system has extremely high identification accuracy.

In the future, we would like to introduce the system to actual production lines and try to apply it to various industrial products in high-mix, low-volume production. The system’s accuracy will also be verified under various lighting conditions to establish a more practical visual inspection system.