Detection of Oil-Containing Dressing on Salad Leaves Using Multispectral Imaging

There is a need for non-invasive methods for nutritional analysis of food that can address the drawbacks of current methods such as color photography, which cannot distinguish between energy-dense and zero calorie foods. This paper discusses a novel, multispectral approach for the identification of oil, particularly in salads, and defines a pseudo-reflectance term. A custom-made multispectral camera was used to collect a novel, publicly shared dataset of images of untreated lettuce leaves or leaves treated with vinegar, oil, or a combination of these. The camera captured image data at 10 wavelengths $\in $ [380nm,980nm] across the electromagnetic spectrum in the visible and NIR (near-infrared) regions. Imaging was done in a lab environment with the presence of ambient light. Mean spectra were extracted from the regions of interest in the multispectral cube and used to compute pseudo-reflectance. ANOVA (Analysis of Variance) was performed to look for variances in the pseudo-reflectance curves. ANOVA proved that the differences between group means of the four treatment groups (oil, vinegar, oil plus vinegar, control) were statistically significant. Pairs of groups showing the greatest significance were established using a Tukey post hoc test. Sequential Forward Selection (SFS) was used to determine 5 optimal feature wavelengths from the whole feature space (410 nm, 455 nm, 485 nm, 810 nm, and 850 nm). A combination of visible (VIS) and infrared (IR) wavelengths, selected using SFS, showed the greatest potential for discrimination between groups containing oil and groups that do not contain oil with a classification accuracy of 84.20%. The pseudo-reflectance values were statistically proven to be sensitive to the presence of oil as a dressing. This research has demonstrated the feasibility of implementing a multispectral imaging technique for identifying the presence of oil in salads and possibly an energy content detection system.


I. INTRODUCTION
Food and dietary assessment in the present day has become a major field of research, demanding modern techniques and automation. Dietary assessment using self-reporting can be replaced by advanced food monitoring systems based on wearable sensors. The possibility of detecting food intake using sensors has been explored earlier [1]- [6]. The sensors can also determine the number of unique food items in a meal [7]- [9]. Energy intake may be estimated from sensor data, such as counts of chews and swallows [10], [11], or hand gestures [12], [13]. The sensors are also capable of monitoring behavioral metrics of food consumption, such as The associate editor coordinating the review of this manuscript and approving it for publication was Lin Wang . the meal microstructure [14]. The sensor-based methods can be extended by using imaging to identify the type of food, portion size and energy contents.
Food image analysis is emerging as a supplement to dietary assessment based on sensor techniques. Food image analysis can provide an accurate tool for monitoring of ingestive behavior by capturing imagery of food intake. Food identification using deep learning methods [15], [16] and portion size estimation models [17], [18] were reported as methods of food image assessment. The progressive trend towards a food image-based dietary assessment has received much focus on several imaging methods that can be employed in dietary assessment. Red-Green-Blue (RGB) color photography, spectroscopy (ultraviolet, visible and/or near-infrared absorption) and spectral imaging are the three popular imaging methods involved in this area of food content analysis. RGB color photography involves recording color information of the scene in the view of the camera using electronic sensors or light-sensitive chemicals. The recorded information is then used to reproduce the original colors by mixing various proportions of red, green and blue. RGB color photography has been used previously for identifying skin defects in citrus fruits [19] and has also been used to replace manual sorting and grading of fruits [20]. Although color photography can be used to monitor food items [21], it can be one-dimensional in terms of visual information when compared to spectroscopy and needs complicated image processing to analyze the images. Instances where image processing algorithms cannot identify the contents of the food items, despite the availability of food imagery (e.g. differentiating zero-calorie vs regular soda), can lead to inaccurate energy intake estimates. Both the human eye and computer vision lack the ability to recognize if a food item is energy dense, such as fats and oils. Another challenge is distinguishing substances that look alike, for example transparent substances such as oil, vinegar and water. Salads with low-calorie or high-calorie dressing may not be discernible by just using the RGB colored food images. These limitations of color photography have critically influenced academic dialogue on inspecting the spectral properties of food items using spectroscopy or spectral imaging.
Spectroscopy, particularly Near-Infrared (NIR) spectroscopy can be useful in food analysis. NIR spectroscopy is routinely used for the compositional analysis of food ingredients and agricultural produce. NIR instruments use infrared light to analyze materials. The main advantage of NIR spectroscopy is the ability to analyze high-moisture food due to the relatively weak absorption of water. Spectroscopy was used to predict soluble solid concentrations over cut sections of kiwifruits [22]. Another advantage is the time required for the analysis, which can be done within minutes. However, spectroscopy is dependent on certain NIR reference standards which are used for calibrating the NIR instruments and can be a setback for the whole experiment if these references are not precise. Also, spectroscopy is complex in terms of the equipment used, which can also be expensive.
Spectral imaging has proven to be an alternative to NIR spectroscopy with two major methods, namely, hyperspectral and multispectral imaging. Spectral imaging is not restricted to solid items and can be effectively used to analyze properties of liquids [23] or internal quality features of agricultural products. Soluble content of blueberries was studied and evaluated to enhance product quality through efficient sorting [24]. Spectral images offer more discriminating power than standard RGB images and using a reference image to make the spectral data invariant to the light source can help achieve comparable classification results in different illumination conditions [25]. Infrared wavelengths are used along with the visible spectrum wavelengths. Certain features that cannot be identified in the visible spectrum can be seen at the IR wavelengths. In a previous study, sugar content of a melon was visualized using NIR single-spectral imaging [26]. The authors first identified the optimal wavelength using a spectrometer.
The type of light source and the imaging unit used determines whether the approach is a hyperspectral or multispectral imaging technique. Hyperspectral imaging could be used to extract spectral properties of objects using thousands of images, each representing a specific wavelength band in a selected spectrum of wavelengths that are employed for the analysis. The spectrum consists of a large number of continuous wavelength bands. Normalized reflectance spectral data using reference and dark images were used to extract information about bruises in strawberries [27]. Skim and nonfat milk powder were analyzed for differences in their reflectance spectra by defining a reference pseudo-absorbance term [28]. Another study, involving hyperspectral imaging for detection of injuries in peaches, used optimal representative wavelengths achieving a 98.5% accuracy of detection [29]. Hyperspectral imaging can improve the accuracy of monitoring when compared to human inspection [30].
Hyperspectral and multispectral techniques differ in the number of spectral bands that are covered in the imaging system. Unlike hyperspectral imaging, multispectral imaging uses fewer bands reducing the time required for processing. Multispectral imaging has anywhere between 3 to 30 narrow (wavelength) band images of the targeted object depending on the selected wavelengths. There is no continuum in the captured spectrum unlike in hyperspectral imaging. Multispectral imaging may cover the ultraviolet, visible light and infrared spectrum. Each multispectral cube is a collection of 2-dimensional (2D) images at different wavelengths.
Multispectral imaging can be cost and time effective when compared to hyperspectral imaging, and its application on food content analysis can be further explored.
In this study, a novel application of multispectral imaging is discussed. This application can distinguish similar-looking foods that cannot be distinguished with RGB color imaging alone. This paper discusses a system that could be used by consumers with dietary restrictions or those who have an interest in the nutritional profile of the foods they consume daily. The developed system produces reflectance curves of the food items it is exposed to, allowing for comparison between the generated curves. Fig. 2. represents the proposed framework. Analysis of Variance (ANOVA) followed by Tukey analysis tests for presence of high-calorie content, particularly oil on lettuce leaves. Sequential Forward Selection (SFS) was used to identify the wavelengths that are significant, and a Quadratic SVM was used to build a model for detecting the presence of high calorie content. The contributions of this paper are: i) the proposed method is faster ii) easy -to use and portable iii) eliminates the need for complex setups when compared to other existing methods iv)the equipment can be used under ambient light and captures a multispectral cube within 20 seconds v) a pseudoreflectance term is defined to make the model invariant to VOLUME 8, 2020 changing illumination conditions vi) a novel publicly available multispectral dataset.
The paper is organized as follows. First, a summary of related works is presented in Section II, then a description of methods is presented in Section III along with information about the equipment. Sections IV-V present the results and discussions of the proposed application.

II. RELATED WORKS
Multispectral images have been used to classify insects that appear indistinguishable to a human observer [31] and to monitor plant physiological status [33]. Multispectral imaging has been used previously to detect healthy and infected samples of citrus fruits [33] and raw french fries [34]. Adulteration and defects in food items and agricultural produce have been analyzed using multispectral imaging technology in packed food [35], fruits [36], [37], beef [38], rice [39] and tomato paste [40]. Crop monitoring and vegetation index calculation have been implemented using spectral imaging [41], where the authors talk about an RGB color imaging plus NIR imaging system that assess the scene for evaluating the health status of crops. [42] and [43] employ a multispectral imaging system on UAVs. [44] summarize the use of a multispectral sensor in the quality assessment of plant foods.
Food quality control and sorting were achieved using multispectral imaging [45]. Food samples can also be tested for specific constituents as in the following studies: water content in beef samples [46], and sugar content in Fuji apples [47].

A. EQUIPMENT
The study utilized a custom multispectral camera built at the University of Alabama. The camera consists of an embedded computer (Raspberry Pi Model 3B) running Raspbian Stretch 4.14 operating system with a Pi NoIR Camera V2 and a custom-made multispectral illumination circuit. The illumination is provided by the light-emitting diodes (LEDs) that are controlled by the Raspberry Pi via power MOSFETs. The LEDs are synchronized with the camera, and the system captures ten continuous images, one for each LED illumination. Each image had a resolution of 8 megapixels. The LED illumination can be turned off to capture images with background illumination. The wavelengths used here are as follows: 385 nm, 410nm, 455 nm, 485nm, 590 nm, 660 nm, 740 nm, 810 nm, 850 nm and 940 nm. Fig.1 depicts a sample sequence of images captured by the multispectral device.

B. DATASET
To generate the dataset of multispectral images, lettuce leaves of different shapes and sizes were picked randomly from a fresh whole head of iceberg lettuce. The size of the leaves was limited to fit in a square of side 10 cm to match the camera field of view. The images were taken in ambient lighting conditions in the laboratory, with a lux meter reading of 355 Lux. Each leaf was placed on a chrome black surface at a distance of 10 cm from the camera.
Each leaf was subjected to a treatment based on the group of testing. There were 4 groups (oil, vinegar, oil plus vinegar, and control) with one of them being the control group without any substance being added. Each substance was added using an atomizer and in such a way that there was no excess of the substance. Generic non-branded vegetable oil and apple cider vinegar were used. The caloric contents of the substances used in this study is as follows: oil-120 calories per serving (1Tbsp), apple cider vinegar-0 calories per serving (1Tbsp) and lettuce 5 calories per serving (1 Cup). Energy content refers to the number of calories in each substance. Oil is an energy-dense substance while vinegar is a zero-calorie substance.
The number of samples (k) in the dataset (sample size) was computed for two factor ANOVA. There were 4 groups in this study and the analysis was conducted at the 0.05 significance level. As suggested in [48], this study is designed to have a power of 0.80. The effect size is chosen to be 0.25 which represents a medium effect size. The power analysis found that the minimum sample size was 44.6 samples per group, therefore, we used 60 samples for each group (k=240). Each sample had two sets of images (one before the treatment and one after the treatment) plus an extra image that was captured under ambient lighting condition that was used as a reference (represented in Fig. 3). The reference image was taken before the treatment. The before and after image sets were multispectral cubes with 10 images for the 10 selected wavelengths in each of them. This collected dataset is publicly available at IEEE-Dataport [49] and Box drive [50].
All the images were cropped into square shaped regions of interest (ROI) of side 2 cm. The ROIs were selected by manual cropping by making sure that the sprayed sample covered the entire or most part of the cropped area. We assume that there is equal coverage of the sample in the ROI. The final dataset had 4940 cropped regions of interest with 240 samples. The procedure to acquire the multispectral dataset is as given below: [1] Capture one image without LED illumination (reference).
[2] Capture multispectral images using the LED illumination.
[3] Apply a small amount of the test substance (no test substance for the control group) and then repeat step 2.

C. PSEUDO-REFLECTANCE
One major contribution of this paper is the introduction of the pseudo-reflectance term. Traditionally, in multispectral imaging a reference standard is used to get the relative reflectance spectra of an object of interest. Diffuse reflectance standard is an example reference that is used for multispectral imaging applications. These references are expensive and can be used to calculate relative reflectance only with spectral images taken in a dark room. Pseudo-reflectance eliminates the need for such references and allows for imaging to take place in ambient lighting. Pseudo-reflectance makes use of two spectral images, one image of the background and one image with the substance of interest placed on the background, and one RGB color image of the background. In this case, the background refers to a single lettuce leaf and the substance refers to a particular treatment. To determine the pseudoreflectance value, each ROI was first converted to grayscale. The average intensity value of all the pixels in the region was calculated for each of the three ROIs -background image, treatment image and reference image as given in (1-3).
where, m, n = dimensions of the cropped ROI I(x, y) = grayscale intensity of the pixel at position (x,y) p (k,λ) = avg background − avg treatment avg reference (4) VOLUME 8, 2020  Pseudo-reflectance (p) at each wavelength λ for each sample k is calculated as in (4) . Fig 4. describes the framework of the proposed approach summarizing the methods used in this study after calculating the pseudo-reflectance.

D. ANALYSIS OF VARIANCE (ANOVA)
ANOVA was performed to identify if there was a significant difference in pseudo-reflectance values within the treatment groups. Prior to performing ANOVA, several assumptions were made about the data and are as follows: • Independence of samples-The samples were independent of each other. The observations were not correlated in time. The observations were not correlated in space.
• Normality -the distributions of the residuals are normal.
• Equality of variances between treatments Two-Factor ANOVA was performed on the pseudoreflectance dataset. Two-factor ANOVA is an extension of the one-way ANOVA and evaluates the influence of two different independent categorical variables on a continuous-dependent variable. It assesses the effect of each independent variable and checks if there is any interaction between the variables. The number of samples in each group is the same. There were 60 samples in each group.

E. TUKEY TEST
The Tukey test is performed to identify which groups are similar to each other and which groups are different to each other with respect to the pseudo-reflectance values. The test conducts pairwise comparisons and checks if the two groups are significantly different.

F. SEQUENTIAL FORWARD SELECTION (SFS)
Sequential forward selection (SFS) method was used for selecting the wavelengths (features). The pseudo-reflectance values were provided as input to the wrapper-SFS algorithm. SFS is a greedy search algorithm that starts from an empty set and sequentially adds the feature x * that maximizes A(Y k + x * ) when combined with the features Y k that have already been selected. SFS begins with zero attributes and evaluates all feature subsets with exactly one feature at a time. The wrapper based forward sequential search selection (SFS) method employs a classifier to select a best feature subset. SFS which uses a quadratic SVM classifier is chosen as the method for feature subset selection. SFS was used to get a best feature subset for the classification of energy-dense and zero-calorie substances (oil and vinegar, respectively). The classification accuracy was used to identify the best subset.

G. SVM CLASSIFIER
This section discusses a classifier that can be used to classify between energy-dense and zero-calorie substances. Once trained this classifier can be used to identify which treatment group the pseudo-reflectance values belong to. An SVM classifier finds the best hyperplane that separates all data points of one class from those of another class. A polynomial SVM of degree 2 (Quadratic) was constructed for the pseudo-reflectance data using 5-fold cross validation.. The value of 5 was chosen to split the data in an 80:20 ratio for training and validation, respectively, in each fold. The SVM model parameters cost (c) and gamma (g) were tuned using parametric sweeps for optimal values of the two parameters. The treatment groups -control and vinegar were considered as zero-calorie substances while the groups-oil and oil plus vinegar were considered as energy-dense. The two classes for the SVM classification were the samples that do not contain oil (control and vinegar) and the samples that contain oil (oil and oil plus vinegar).
The SVM model is evaluated using the accuracy of classification. Accuracy of classification (A) is given by (5).
where, tp: number of true positives of class high-calorie, tn: number of true negatives of class high-calorie, fp: number of false positives of class high-calorie, fn: number of false negatives of class high-calorie.

IV. RESULTS
The ANOVA results are summarized in Table 1. There was a significant difference in the means for the different treatments unlike the case for different wavelengths. Table 2 summarizes the results of the Tukey test. It was seen that the groups without any treatment and with the vinegar treatment did not have any significant differences. The groups with the treatment with oil and oil plus vinegar are significantly different from each other. SFS identified 5 optimal feature wavelengths ranked in the following order (more prominent to less prominent): 455nm, 485nm, 850nm, 810nm and 410nm. The implementation of the SFS algorithm is described in figures 5 and 6 (appendix A).
SVM classifiers for the 4 feature subsets were optimized through parametric sweeps for gamma and cost. Tables 5, 6, 7 and 8 (appendix B) summarize the parametric sweeps.
The SVM results using the 4 feature subsets are summarized in Table 3. Table 4 represents the confusion matrix for the model using the SFS feature subset. The comparison using different feature subsets was made to identify the importance of the each one of the bands in the multispectral application. It was seen that using the wavelength groups, individually, resulted in lower accuracies than by using a mixture of wavelengths from both wavelength bands (VIS and IR).

V. DISCUSSION
The study discussed a novel application of multispectral imaging. The pseudo-reflectance values were sensitive to the presence of an energy-dense substance (oil).
The Tukey test showed that the means of pseudoreflectance values for the control group and the treatment  group with vinegar were not significantly different. The treatment with oil and the treatment with oil plus vinegar was found to be significantly different with respect to each other and significantly different from the other two groups. The results can be interpreted in terms of the energy content of the treatments. Vinegar is a low-calorie substance, and oil is a high-calorie substance. The results suggest that it might be possible to detect the presence of a high-calorie substance, in this case oil.
It was seen that the treatment group with oil plus vinegar was not only different from the groups that did not contain oil, but it was significantly different from the treatment group that contained oil alone. This might be due to the fact that a mixture of oil and vinegar is not a stable emulsion. The emulsion separates and combines randomly, which is why the pseudo-reflectance for this treatment matches with neither vinegar nor oil. Nonetheless, the study showed that it was possible to distinguish between the groups containing oil and groups that did not contain oil.
This study was conducted only on a particular type of salad leaves, namely iceberg lettuce, and in a single leafbased experimentation. The study could be further extended to other types of ingredients and to a whole bowl of salad with several ingredients. Also, the salad toppings were restricted to oil and vinegar. Other salad dressings could be tested for their presence and analyzed using multispectral imaging in the future.
The contributions of the proposed method are severali) The imaging was done in ambient, wild conditions unlike in several other multispectral studies that involved setting up a specific environment such as a dark room. Wild refers to free-living conditions. Dark rooms are typically used for spectral imaging so that external light sources do not affect VOLUME 8, 2020  the imaging. The use of pseudo-reflectance term in this paper eliminates the need for such setups. ii) This study shows a cost-effective and easy-to-use method of conducting the multispectral analysis. iii) Pseudo-reflectance could be used to evaluate the scene for food assessment. This term also eliminates the need for using a reflectance standard as a  reference correction of light source effects, as in [24]. iv) The current study was conducted in an indoor laboratory under lighting conditions which mimic typical-indoor food intake cases, such as restaurants and homes. However, this method was not tested in an outdoor setting which may or may not influence the results. This could be something that could be explored in the future. V) The novel multispectral dataset is made available to the public.
The SVM classifier achieved a good reasonable accuracy of classification. This classifier can be used to distinguish between energy-dense and zero calorie foods enabling one to detect if substances like oil were in fact added to a food item, in this case a salad.
The classification accuracies using all wavelengths, VIS only and IR only were 85%, 81.70% and 79.20 % respectively. It was seen that by using the optimal (SFS) wavelengths, five wavelengths from a total of ten wavelengths, a classification accuracy of 84.20 % could be achieved. A combination of NIR and VIS wavelengths produces an acceptable level of accuracy suggesting that multispectral analysis could indeed open new possibilities in dietary assessment. Additional information in the NIR space could lead to better analysis and understanding of the content in the scene of capture. Since the number of wavelengths supported by the equipment used here is limited, the user is limited to this set of wavelengths. Higher classification accuracies could possibly be achieved if more wavelengths are included. It was seen that the model with NIR wavelengths performed as good as the model with visible wavelengths in terms of accuracy. Using additional NIR wavelengths could further facilitate the exploration of NIR spectral image-based detection and classification of the high-calorie content in salad leaves. Selecting the optimal wavelengths using initial trials and pilot studies could help to minimize the time of capture and the number of light sources on the equipment. The classifier performs poorly for the treatment group with oil plus vinegar. As discussed earlier, this might be due to the characteristics of the mixture containing oil and vinegar.
Another limitation of the paper is that the methods do not predict the actual caloric values of the content. This could be a potential future course of work which involves the determination of the caloric value by using regression or other suitable methods.
The application discussed in this paper could be used to monitor and keep track of the food content that is consumed by humans. The method could be potentially used by nutritionists to identify if the food items are nutrient-dense or consist of empty calories. The multispectral data can also be provided as an input to computer vision-based deep learning models for dietary assessment.

VI. CONCLUSION
In this study, a novel method for the identification of high calorie content particularly in salads was proposed. It was seen that a combination of visible light wavelengths and near-infrared wavelengths was optimal in detecting oil in the lettuce leaves. A pseudo-reflectance term was defined and analyzed. This paper contributed a method that could be implemented in the wild without any controlled experiment setups and introduced a pseudo-reflectance term. The pseudoreflectance values from the device were statistically proven to be sensitive to the presence of oil. The classification accuracy for oil content was 84.20%. This study proposes a potential of using multispectral imaging for the detection of oil containing dressing and possibly an extension to other food categories. Future-work could involve a complex scene setting such as a bowl of salad and other varieties of salad dressings.  Tables 5-8. [50] V. Raju  EDWARD SAZONOV (Senior Member, IEEE) received the Diploma degree in systems engineering from the Khabarovsk State University of Technology, Russia, in 1993, and the Ph.D. degree in computer engineering from West Virginia University, Morgantown, WV, USA, in 2002. He is currently a Professor with the Department of Electrical and Computer Engineering, The University of Alabama, Tuscaloosa, AL, USA, and the Head of the Computer Laboratory of Ambient and Wearable Systems. His research interests include wireless, ambient, and wearable devices, and methods of biomedical signal processing, and pattern recognition. In his Laboratory, he developed devices such as a wearable sensor for objective detection and characterization of food intake; a highly accurate physical activity and gait monitor integrated into a shoe insole; a wearable sensor system for monitoring of cigarette smoking; and others. His research has been supported by the National Science Foundation, the National Institutes of Health, the National Academies of Science, and by state agencies and private industry and foundations.