DoFP-ML: A Machine Learning Approach to Food Quality Monitoring Using a DoFP Polarization Image Sensor

Good nutrition is an important part of leading a healthy lifestyle. This has brought into stark focus the need for efficient and low-cost methods for large scale food quality assessment. This article proposes a non-invasive and non-destructive system for estimating the freshness of apples using polarization images from a Division-of-Focal-Plane (DoFP) polarization camera. The proposed system uses Machine Learning Systems namely, Support Vector Regression (SVR) and Gaussian Process Regression (GPR), to estimate the age of apples and determine if they are fit for consumption even before the external rot appears on the fruit. Initially, the reconstructed images namely, Degree of Linear Polarization (DoLP) and Angle of Polarization (AoP), are generated from the polarization image and their respective correlations with the actual age of apples (in days) are established. These reconstructed images are then fed as input features to the Machine Learning Systems to ultimately estimate the age of the apples. Experiments on real data obtained from the DoFP camera show that the proposed system is non-destructive and capable of non-invasively estimating the age of the apple with an average accuracy of up to 92.57%.


I. INTRODUCTION
Food quality and freshness assessment is increasingly becoming an area of interest for both the consumers and the food processing industries. Food quality plays a vital role in decision making regarding the storage and processing requirements. The periodic chemical and biological analysis using techniques such as chromatography, spectrophotometry, and electrophoresis among others, are conventionally used in evaluating the food quality. These methods, however, are costly, time-consuming and require a trained operator. The conventional methods are also destructive in the sense that they destroy the tested sample, resulting in financial losses, when performed at a large scale. Some nondestructive The associate editor coordinating the review of this manuscript and approving it for publication was Ting Wang . monitoring techniques such as optical spectroscopy, acoustic measurements, and chemical profile monitoring using electronic noses, have already been used for affordable and rapid monitoring [1]. Acoustic methods are capable of measuring the elasticity of tissues and can therefore be used to measure softening of apples with increased storage time. Optical techniques are the most popular of the non-destructive techniques because of their simplicity in implementation. They involve successive measurements of a sample without causing damage to the sample. These optical techniques are often based on a specific property of light [2].
One such property of light is polarization. This property, although imperceptible to the human eye, has attracted significant attention as it is shown to provide very useful information unavailable in intensity/wavelength imaging domains [3]. This is why polarization imaging systems have been employed in various applications including biomedical imaging [4], [5], material classification [6], [7], and 3D shape reconstruction [8].
In [9], the polarization property of light was used to study food properties. The paper introduced an experimental setup based on Division-of-Time (DoT) polarization architecture that comprises a standard CMOS camera and a pair of polarization filters to take polarization images of apples over a period of time. A polarized light, generated using a linear polarizer in front of a white light source, was incident on the fruit and the phase information of the reflected light was recorded by the standard CMOS camera, which has a linear polarizer placed in front of it. The Freshness of fruits was thereafter correlated with the changes in the phase information of the reflected light. To monitor the changes, Stokes' Degree of Polarization (DoP) and Degree of Linear Polarization (DoLP) were utilized. This technique was complex in terms of hardware as two polarization filters and a standard CMOS camera were required. Furthermore, a perfectly linearly polarized light was assumed which made the subsequent determination of DoLP incomplete.
In [10], polarization was also used for quality monitoring of apples. While the setup was somewhat similar to [9], the paper showed that by eliminating the linear polarizer in front of the light source, better results could be achieved. The method to determine the DoLP was also easier to compute. While this method had less hardware complexity, the DoLP calculations were less accurate than the calculations in [9].
The recent advances in solid state technology has spewed out a special class of polarization sensors known as Divisionof-Focal-Plane (DoFP) polarization image sensors. These highly compact sensors have the micro-polarizer array fabricated directly on top of the image sensor. The micro-polarizer filter array contains linear polarizers offset by 45°arranged in a 2×2 periodic pixel pattern, called a ''super-pixel'', as shown in Figure 1. This periodic structural pattern of the super-pixel resembles the Bayer Color Filter Array (CFA) pattern in color image sensors [11]. A huge advantage of the DoFP architecture over the Division-of-Time (DoT) architecture is the ability to capture the full polarization information of a target object in a single frame. This in turn makes the determined DoLP more complete and therefore, more accurate. In this work, we present a non-destructive and non-invasive automated system (DoFP-ML) for estimating the freshness and quality of apples in terms of age. Unlike the techniques in [9] and [10], the proposed DoFP-ML system uses a Division-of-Focal-Plane (DoFP) polarization camera to capture the change in polarization property of apples over time. The contribution of this work is twofold: Firstly, through the use of a DoFP polarization camera, the full polarization information of the apple is recorded in a single frame and thus, subsequent determination of DoLP is shown to be more accurate. Secondly, we introduce machine learning for automated prediction of the age of apples. While the systems presented in [9] and [10] correlated DoP and DoLP to ripening and decay of apples, the exact age of the apple was not demonstrated. Our proposed method, on the other hand, is able to estimate the age of the apple, with acceptable accuracy, by using DoLP and AoP as input features to a machine learning system.
The rest of the paper is organized as follows: Section II presents the proposed DoFP-ML system. Experiments' results are discussed in section III while conclusions are drawn in section IV.

II. DoFP-ML SYSTEM
The DoFP-ML system aims to predict the freshness of apples in terms of age (number of days) at room temperature. The block diagram of the proposed DoFP-ML system is presented in Figure 2 while the description of the building blocks is as follows:

A. IMAGE ACQUISITION
The polarization images were captured using the ''4D Polar-Cam snapshot micro-polarizer camera'', which is a DoFP polarization camera. As illustrated in Figure 3, a white light from a LED array is incident on the apple and the reflected rays are recorded by the DoFP camera. The proposed system takes full advantage of the micro-polarizer array structure to record the full polarization information of the reflected light in a single frame. The captured polarization image of the apple is shown in Figure 4(a).

B. FRAME REMOVAL
In order to obtain a standard size image and to facilitate the segmentation process, the black frame around the image is removed, as shown in Figure 4(b), while ensuring that the polarization pattern remains undisturbed.

C. DEMOSAICKING
The image obtained using a DoFP camera is a mosaic image composite of four low-resolution sub-images (I 0°, I 45°, I 90°, and I 135°) . These low-resolution sub-images are extracted, and their respective full-resolution images are generated using Interpolation algorithms, as shown in Figure 5. Interpolation algorithms are to DoFP images what demosaicking algorithms are to images from color imagers based on Bayer CFA. This is why they are sometimes referred to as ''polarization demosaicking algorithms''. In this work, the selected interpolation algorithm is the nearest neighbor interpolation algorithm because of its simplicity. The nearest neighbor interpolation technique involves replacing the missing pixel value by its nearest neighbor within a 3 × 3 block [12].    A recursive application of this technique will ultimately generate the respective four full-resolution images.

D. SEGMENTATION
Segmentation is the process of dividing a digital image into multiple segments (sets of pixels). It can also be thought of as the process of assigning a label to every pixel in an image such that pixels with the same label share certain visual characteristics [13]. In this work, segmentation using Otsu's method [14] is applied on one of the full resolution subimages. This results in a binary image (Mask), with white color (digital 1's) corresponding to the apple and black color corresponding to the background as shown in Figure 6(a). Each of the full resolution images is then multiplied (pixel by pixel) with the mask resulting in zeroing of all pixels in each image except those corresponding to the apple. One of the resulting images after the segmentation stage is presented in Figure 6(b).

E. FEATURES EXTRACTION
The polarimetric property of light are represented using Stokes parameters (S 0 , S 1 , S 2 , and S 3 ). The Stokes parameters are derived mathematically using the four full resolution intensity images as follows: As the DoFP imager inherently captures linear polarization, the S 3 term, which is the difference between the Right Circular Polarization (RCP) component and the Left Circular Polarization (LCP) component, is hereby ignored.
From the Stokes parameters, other images that have more physical meanings can be reconstructed [15]. These are Degree of Linear Polarization (DoLP) and Angle of Polarization (AoP) images, which are derived as: These reconstructed images will serve as input features to the Machine Learning Block.

F. MACHINE LEARNING
Our intention is to learn a model function f (.) that can be used for estimating the age of apples using machine learning based regression. This is implemented in two phases, namely, the training phase and the testing phase. During the training phase, apple features (DoLP and/or AoP) collected in the past (from training data set) along with the associated apple age, are fed to the machine learning block to model the function f (.). During the testing phase (running phase), the trained model f (.) which takes the features of new apples (DoLP and/or AoP) as input, is used to predict the age of these new apples. Figure 7 shows the block diagram of operation of the two phases. In this work, two machine learning algorithms, namely, Support Vector Machine (SVM) and Gaussian Process (GP) method, are implemented. Since the objective is to predict the age of an apple to check whether it is still valid for consumption, both algorithms are used in the regression mode.
Support Vector Regression (SVR) is a regression method that is quite convenient for this application. It is a supervised machine learning method that fits a model for the data in the feature space within deviations from the target values. For complex models, it uses kernel functions to map the problem in the input space to a higher dimensional feature space where regression problems that are highly nonlinear in the input space become linear in the higher dimensional space.
To reach the desired model, it implements the structural risk minimization principle [16] to minimize the risk within Vapnik's -intensive loss function. A detailed formulation of the SVR method can be found in [17].
Gaussian Process Regression (GPR) is a non-parametric Bayesian approach used in regression problems [18]. It is a kernel-based supervised machine learning method that assumes a prior probability distribution of the data under consideration. The prior is updated based on the training data to generate a posterior probability distribution that is completely described by its covariance and mean value. It is this mean value of the posterior function that can be used for prediction [19], [20]. GP defines a probability distribution over possible functions [21].
A key assumption in GP modelling is that our data can be represented as a sample from a multivariate Gaussian distribution [22]. An important property of Gaussian distributions is that conditioning and marginalization operations result in normal distribution functions. Initially, the regression function Y = f (X ) is defined as a multivariate Gaussian distribution function representing the prior distribution. Based on the Bayesian inference approach, the prior is updated using the training data to produce the posterior distribution which will be Gaussian as well. The mean of the posterior represent our regression result. In order to account for noise and errors inherent in the training data, a noise term is added to f (X ) leading to Y = f (X ) + ω, where ω → N (0, σ 2 ). A detailed mathematical formulation of GPR can be found in [18].
In both machine learning systems, the kernel function is ''RBF kernel''. 80% of the data is used to train the system with 5-fold cross validation implemented to optimize the kernel parameters. The remaining 20% of the data is used to test the system performance. In order to judge the system's ability to correctly estimate the age of apples from polarization images, the estimated ages are compared with the real ages using Mean Absolute Error (MAE). The output of the algorithm is passed through a floor function which yields integer values corresponding to age estimations in days.

III. EXPERIMENTAL RESULTS AND DISCUSSION
To validate the proposed DoFP-ML system, an experiment was conducted on five ''Royal Gala'' apples. Under room temperature, these apples were observed until the exterior rot became visible to the naked eye. The DoFP camera was used to capture the images of the apples every other day for a period of 16 days (first day is day 0). This amounts to nine ''shooting'' days where images were captured. On each of the 9 days, each apple has its polarization image taken 5 times, making a total of 25 images per day and a cumulative total of 225 images at day 16. The five images recorded on each day for the respective apples went through pixel-wise averaging to remove the effect of inherent random noise. The respective Stokes parameters as well as the reconstructed images for each apple were then determined after undergoing the processes outlined in section II. The first study will be determining the best features to be fed into the machine learning systems.

A. FEATURES DETERMINATION
The matrices of the reconstructed images (DoLP and AoP) are available at this stage. For analysis, the respective mean values of these matrices are considered to ascertain their relationship with the freshness of the apples. While all the apples showed a similar trend, only the variation of average DoLP of ''Apple 5'' with increasing storage time is demonstrated in Figure 8(a). As can be observed from the data  points, an inverse or negative relationship exists. Furthermore, the data points are seen to reside very close to the best fit line, which is indicative of a strong negative linear relationship. This is contrary to the relationship established in [9] where the DoLP is shown to rise with increasing storage time. This discrepancy stems from the calculation of DoLP from Stokes parameters. In [9], a perfectly linearly polarized light was assumed and therefore, only the horizontal and vertical components of polarization were considered. Therefore, the S 2 term in Equation 5 was ignored when calculating DoLP. However, in reality, light is never perfectly linearly polarized [15]. By using the DoFP imager, the polarization information of the reflected light through the 45°and 135°d irections are also recorded. This is due to the inherent ability of DoFP polarimeters to record polarization information along the four directions (0°45°, 90°and 135°) in a single frame. With this additional polarization information, the calculated DoLP is more complete and hence, more accurate.
To accentuate the reliability of the negative trend of DoLP, visual results are presented in Figure 9. The background is made to be uniformly red so that only the DoLP results of the actual apple can be analyzed using the colorbar on the side. The visual results show that with increasing storage time, areas of high DoLP values (blue and green) diminish, while areas of low DoLP values (yellow and red) start to become prominent. This decrease in visual DoLP values is therefore consistent with the analytical results shown in Figure 8(a) to affirm an inverse or negative relationship between the average DoLP and age of apples, making it a suitable parameter to monitor and predict the age of apples.
The relationship between the second reconstructed image (AoP) and the age of apples is then studied. The average AoP is plotted against time (days) in Figure 8(b), where the data points are observed to exhibit no discernible pattern. The data points highly fluctuate around the best fit line, which points to a very weak linear relationship between the average AoP and the age. Also, the visual results of the average AoP in Figure 9 do not show any definitive pattern which is in agreement with the analytical results in Figure 8(b). Therefore the average AoP appears not to be a reliable parameter to monitor and predict the age of apples.

B. MACHINE LEARNING
So far, two parameters, the average DoLP and the average AoP, have been studied and related to the age (freshness) of the apple. In the training phase, one or both of these two parameters are fed into the system along with the actual age of the apple. The training phase aims to find a function f (.) that models the relationship of Average DoLP and/or Average AoP with the age of the apple. The modeled function f is then used in the testing phase to estimate the age of the test apple.
The dataset used in this work consist of 225 DoFP images for 5 five apples. 80% of the dataset (180 images corresponding to four apples) is utilized to train and validate the system using 5-fold cross validation method. The remaining 20% (45 images corresponding to the fifth apple) is used test the performance of the system. This was repeated 5 times so each apple gets a turn to be the test apple. The features 'average DoLP' and 'average AoP' take turns in being the input features, with a final round having both features as input-pair fed into the system. The machine learning systems to be used will be Support Vector Regression (SVR) and Gaussian Process Regression (GPR). Performance of the systems is verified by calculating the overall Mean Absolute Error (MAE) as well as the estimation accuracy.

1) SUPPORT VECTOR REGRESSION (SVR)
In this experiment, the machine learning system used is SVR. A floor function employed after the regression block results in discretization of the output into integer days and thus, the system becomes a multiclass classifier.
In Table 1, the results of the experiment by varying the input features are presented. In addition to the overall MAE, the accuracy of the classifier within a 2-day and 4-day error tolerance is presented. From Table 1, it can be seen that using the two features as an input-pair, instead of either of them, resulted in lower overall MAE and a higher classification accuracy. Interestingly, the system when AoP was used as a single input feature performed better than when DoLP was the single input feature. This inconsistency with the results in Figure 8 is justified by the kernel trick [23], [24]. A kernel-based machine learning algorithm uses a kernel function to map the problem in the input space to a higher dimensional space where complicated classification problems become linearly separable, and regression problems that are highly nonlinear in the input space become linear in the higher dimensional space. Therefore, SVR becomes capable of fitting a nonlinear regression function in the input space, which resulted in a superior performance of the system when AoP was the input feature, as opposed to when DoLP was used alone.
For a more detailed illustration of the SVR system results, Figure 10 presents the relationship between the actual age of the apples and the Mean Absolute Error in estimating the apple's age. For the case of the input-pair features (DoLP and AoP), the error is seen not to exceed 2.1 days until day 12. The error slightly exceeds 3 at day 14 and reaches 4.5 at day 16, the day when the exterior rot became visible. Within the first 12 days, the maximum error is 2.1 days which corresponds to a maximum percentage error of 13.125% or minimum age estimation accuracy of 86.875%. During this period, the average error is 1.41 days which represents a percentage error of 8.86% or an average age estimation accuracy of 91.14%. Considering the minimum accuracy case (i.e maximum error of 2 days), and taking day 11 as the threshold above which an apple is considered to be unhealthy for consumption, the apples that are actually older than 12 days will still classify above the day 11 threshold and consequently, be deemed unhealthy for consumption. Therefore, it can be said that during the first 10 days, the system is capable of accurately estimating the apple's age, within a 2-day error tolerance. After this period, the system labels the apple as unhealthy for consumption.
For the DoLP case in Figure 10, it is seen that the error fluctuates between 1.3 and 3 until day 12, and then increases sharply to 5 days at day 14, which is worse than the performance of the input-pair features. On the other hand, the AoP case shows continuous fluctuation between 0.5 and 3.5 for the whole range of the experiment, which makes it unreliable for estimating the age of the apples.
Overall, it can be stated that the proposed method when DoLP and AoP are used as input-pair features, can noninvasively estimate the age of the apple and detect freshness with acceptable accuracy, before external rot appears.

2) GAUSSIAN PROCESS REGRESSION (GPR)
In this study, the machine learning system is switched to GPR while all other factors remain unchanged. The results from the GPR system are shown in Table 2.
From Table 2, it can be seen that using both DoLP and AoP as input-pair features to the GPR system resulted in an overall accuracy of 79.6% within a 2-day error tolerance, and 92.9% within a 4-day error tolerance. Similar to the SVR system, the accuracy of the GPR system was higher when the input-pair features (DoLP and AoP) were used than when either of them was used as a single input feature. Once again, the performance of the system being higher when AoP was used as a single input feature than when DoLP was used, is justified by the kernel trick [23], [24].
In Figure 11, the relationship between the actual age of the apples and the mean absolute error in estimating the apple's age is illustrated. It can be clearly seen that for the case of the input-pair features (DoLP and AoP), the error in estimating the age of the apples remains below 2 days until day 13. It reaches 2.5 at day 14 and 3.7 at day 16, the day when the exterior rot starts to appear. The maximum error within the first 12 days of the experiment is 2 days which corresponds to a maximum percentage error of 12.5% or minimum age estimation accuracy of 87.5%. The average error within this period is 1.19 days which represents a percentage error of 7.43% or an average age estimation accuracy of 92.57%. Considering the minimum accuracy case (i.e maximum error of 2 days) and taking day 11 as the threshold beyond which an apple is considered to be unhealthy to eat, the apples that are actually older than 12 days (with the inherent classification error included) will still classify of age above the day 11 threshold, and will therefore be deemed as unhealthy for consumption. Accordingly, the system is said to be capable of correctly estimating the age of the apple (within a 2-day tolerance) during the first 10 days, while after this period, it labels the apple as unhealthy for consumption.
Referring back to Figure 11, it can be seen that for the DoLP case, the error fluctuates between 1.3 and 3 until day 12, and then increases sharply to 5 at day 14, which is worse than the performance of the input-pair features. On the other hand, the AoP case shows continuous fluctuation between 0.2 and 3.4 for the whole range of the experiment which, as in the case of SVR, makes it unreliable for estimating the age of apples. As a result, it can be said that the proposed method with DoLP and AoP taken as input-pair features can non-invasively estimate the age of apples with acceptable accuracy, to determine if it is fit for consumption before the rot and damage signs become visible to the human eye.  A performance comparison of the two Machine Learning systems is presented in Table 3, which shows GPR having a slightly higher average apple age estimation accuracy within the first 12 days. This is in agreement with the results in Table 1 and Table 2 as well as Figure 10 and Figure 11, in showing how the performance of the GPR system exceeds that of the SVR system. Beyond the threshold of day 11, both Machine Learning Systems performed well in labeling an apple as unhealthy for consumption.

C. PERFORMANCE COMPARISON
A comparison of the proposed DoFP-ML method and other noninvasive polarization-based methods reported in [9] and [10] is presented in Table 4.
The first comparison metric is the polarization imaging architecture itself. The methods in [9] and [10] both used the Division-of-Time (DoT) architecture while the proposed DoFP-ML method is based on the DoFP architecture. The advantage of the DoFP architecture is that only one image needs to be recorded for all the analysis. This single image contains the polarization information across all the four polarization directions, because of the micro-polarizer array structure in DoFP imagers. By having a more complete polarization information, the accuracy of DoLP calculation is enhanced, and ultimately a more accurate relationship between the change of DoLP and time (days) is established. Conversely, the systems in [9] and [10] need at least two images for analysis. Moreover, the calculation of DoLP presented is less accurate because some of the polarization components were ignored.
In terms of hardware complexity, the proposed DoFP-ML system is the least complex. The proposed DoFP-ML system requires only a DoFP camera. The system in [9] needs two polarization filters and a CMOS camera. While in [10], their least complex setup still required one linear polarizer and a CCD camera. The hardware complexity is also related to calibration requirement. Intuitively, a very complex system is expected to require a high amount of calibration. The DoFP-ML system with a DoFP camera has low calibration requirement because the micro-polarizer array is fabricated directly on the imaging sensor. This enables the sensor to accurately record polarization information along the four polarization angles. However, in the other systems, the polarization filters need to be rotated after each image is captured. Therefore, very careful calibration is required to ensure the filter is rotated to the desired polarization angle.
Last but not least, the DoFP-ML system introduces Machine Learning to predict the actual age of apples. While the other systems can detect freshness, the actual age of the fruit is not specified. The machine learning block in DoFP-ML however, makes that possible.

IV. CONCLUSION
In this article, we have proposed a non-invasive and nondestructive DoFP-ML system for estimating the freshness of apples using polarization imaging and machine learning. To capture the polarization images of apples, a Divisionof-Focal-Plane (DoFP) polarization camera is utilized. From the captured polarization images, Degree of Linear Polarization (DoLP) and Angle of Polarization (AoP) images are reconstructed. These reconstructed images are then fed into a machine learning block to estimate the age of apples in terms of days. Experiments on real data showed that the proposed DoFP-ML system is capable of estimating the age of apples with an accuracy of up to 92.57%. The proposed DoFP-ML system can help to detect apples that are unhealthy for consumption even before the external rot appears. In the future, we intend to study the possibility of using the proposed system on other fruits and vegetables, with the aim of generalizing it into a comprehensive non-invasive and nondestructive solution for determining the shelf life of food items. This will help big stores to properly manage their stored food items.