Computational Cosmetic Quality Assessment of Human Hair in Low Magnifications

We take advantage of human hair-specific geometry to visualize sparse submicron and micron-sized cuticle peelings with imaging dark-field scattering at highly oblique tip-side illumination. The paper shows that the statistics of these features can directly estimate hair quality is much lower magnifications (down to $20\times $ ) with less powerful objectives when the features themselves are significantly below the system resolution. Our technique for quality categorization of black, blond, and grey human scalp hair samples is successful in detecting healthy and damaged hair in all cases by a large margin (factor of 5 contrast in proposed metric). As demonstrated, the proposed metric even has a strong correlation with the type of damage such as ironing, discoloration, and UV (ultraviolet) exposure. Therefore, this technique has a strong potential for lower cost, portable, and automatic hair diagnostic apparatuses.


I. INTRODUCTION
Hair composition can be studied with conventional polarimetry techniques, SEM (Scanning Electron Microscope) and different spectroscopic techniques [1]- [9] for clinical purposes. Hair quality can be classified thoroughly by extensive spectroscopy and analysis of the essential chemical components in hair down to molecular level [10]- [13]. The infrared and visible scattering of the hair is also recently shown to correlate with hair quality [12], [13]. The current instrument for analyzing hair quality are rather bulky and relatively expensive [6]- [9], [12], [13]. Such a cost and form factors are unjustifiable for cosmetic applications targeted toward portable consumer products and mobile diagnostics. Also, analysis of the chemical compounds of hair is not always necessary as the effects of some of the invasive damaging factors are manifested in the geometry of the hair [4], [9]. Unfortunately, the variations in the features of the cuticles of healthy and unhealthy (for example UV (ultraviolet) exposed or bleached [9]) hair are rather small (submicron) in xy plane (sample plane) and close or below the diffraction limit of optical microscopy. These are three dimensional The associate editor coordinating the review of this manuscript and approving it for publication was Sudhakar Radhakrishnan.
features that can have larger z variations that are not necessarily visible in simple wide-field microscopy configurations. Thus, higher magnification objectives with large numerical aperture (e.g. 100×and 0.9NA) and more complex interferometric or confocal techniques are necessary to resolve and study these features [2], [3], [5], [6]. This higher resolution is obtained at the cost of reduced field of view, reduced depth of field, increased alignment sensitivity, and higher complexity. Other challenges for using optical microscopy while studying human hair are the strong absorption and scattering of light by inner structures of black hair (cortex and medulla - Fig. 1(a)) in transmission mode and strong reflection of light from outer layers of blonde or grey hair ( Fig. 1(b)) in reflection mode.
In recent years, computational methods have eased the demand on the hardware; these methods can reduce the number of acquisitions [14], demix signals at lower SNR [15], and enable imaging in exotic geometries [16]- [18]. Based on prior knowledge of the sample type, by customizing the detection process both at hardware level and software level the detection efficiency can be significantly increased in different applications [19], [20]. Here, we have taken the idea of oblique illumination microscopy also known as dark-field microscopy [8] to the extreme by employing near perpendicular (highly oblique illumination) to capture dark-field scattering of the mechanical features on the surface of the human hair. By exploiting the sparsity of the submicron features and polarization of scattered light, we have been able to characterize human scalp hair with 20× objective at highly oblique LED (Light-emitting diode) illumination. Usually, the quality of human hair is measured with SEM and/or spectroscopic techniques qualitatively (based on cosmetic features) and quantitatively. In qualitative analysis, cracks and peeling cuticles are determined by visual inspection. In quantitative analysis, SEM has been employed to determine human hair quality by measuring the tensile strength (surface tensions, force measurements) [21], [22]. Another method which has been employed to determine hair quality analytically is by using surface with step heights using atomic force microscopy [23]. These methods need high-cost equipment for measurement of strength or height.
In our method, after acquiring hair samples, region of interest (ROI) and features are extracted computationally at low magnifications. The main idea is to compute a damage level for each hair sample based on combination of DT (Delaunay triangle), HD (histogram density), and VD (Voronoi diagram) of these features. Our method can categorize black, blond, and grey hair into healthy and unhealthy (or damaged) with 100% success within our limited (6 strains per category) sample set. This paves the way for lower cost, portable form factors such as cell phone add-ons [24] that could analyze hair quality for cosmetic or sanitary applications.

II. SETUP AND IMAGE ACQUISITION
The setup consists of an infinity corrected objective (20× Olympus UPLSAPO Objective, 0.75 NA, 0.6 mm WD) that is coupled into an aperture, a rotatable linear polarizer, and then a 1.3 Mega pixel camera (Point grey FL3-U3-32S2C-CS). A hair strand is held vertically in midair with a sample holder and illuminated on a black absorptive background with a broadband non-polarized LED in highly oblique angle (85 • ) as in Fig. 2. The sample holder is coupled into a course 0.5" miniature dovetail mechanical 3D stage. The setup is not alignment sensitive due to low magnification of the objective. The LED has a polymer coupled lens and there is no more condenser optics used for the illumination. Due to specific geometry of hair, the difference between healthy and damaged hair becomes significantly evident when a hair strand is illuminated from the tip side. This is because there are a greater number of cuticles that are peeled off from the hair strand. The edges of these cuticles will scatter light that is strongly polarized in the horizontal direction compared to the weaker reflection from the body of the hair strand ( Fig. 2(a)). This helps us to eliminate the background (shown with ⊥ in Fig. 2). We also didn't find as significant of a scattering when we illuminated the sample with the same angle from the root side. This is expected as the peelings have small angle with the main axis of the hair strand ( Fig. 2(b)). The optical axis of the objective and hair shaft are perpendicular to each other and can define a plane of incidence in which the reflected (scattered) light would have S polarization. The consistency of the lighting can affect the observed image; we used a collimated 1 cm by 1 cm white diode to illuminate the sample uniformly across the entire depth of field of the objective and even beyond it. Since the angle of incidence is very large, it is important that the other components such as sample holder don't cast a shadow on the sample. Also the tilt of the sample holder can affect the results. For each sample there is a need for slight alignment to bring the image into focus. We attempted to obtain an alignment where highest number of features are visible and images are in sharpest focus. It is worth mentioning that the system is not strongly alignment sensitive in terms of statistics that it extracts from the samples (the sample can move few hundreds of microns and still be in focus). Misalignment or different positions of same samples generates different data; however, the relative ratio in numbers between damaged hair and unimpacted hair is not changed notably. This robustness to alignment is because the objective has lower magnifications which provides large depth of field for the lens allowing the hot spots to be in focus. Also, the sample holder is designed in such a way that it eliminates ambient light and provides an absorptive black background to the sample. The oblique illumination creates dark scattering of the submicron peelings on the hair which would not be visible unless there is submicron level depth resolution for the acquisition device. This is a significant advantage as it allows the features to be visible with lower magnification optics that are much more cost effective and can capture larger field of view to reveal the statistics of the cuticle peelings.

III. COMPUTATIONAL CATEGORIZATION
The hair categorization procedure is illustrated in Fig. 3. The procedure consists of three major blocks: feature contrasting (optical and computational), quality evaluation, and finally FIGURE 2. Measurement setup schematics: (a) illumination from the tip side results in large number of spot-like features that can be detected with 20× objective. The scattering from the edges of the peeled cuticles is highly polarized as seen in the measurements on the right. (b) Illumination of the same sample from the follicle side. Only a few larger irregularities can be seen. (⊥ is cross polarized with cuticle scattering, is parallel polarization with these scatterings). decision making on category of the hair damage. While other methods can be used in each block, the procedure core remains unchanged.
The hair features are highlighted optically by oblique illumination ( Fig. 4(a)). The acquired hair image is used for input image I as shown in Fig. 4(a).
Statistical metrics are the proper option for evaluating the hair quality, since the method needs to tune to mechanical features of the hair surface below the spatial resolution of the objective. The metric should indicate or at least have a meaningful correlation with the actual damage type or damage extent of the hair sample. In addition, it should work for large variety of hair types such as black, blond, and grey hair without the need for significant parametric initialization. This motivates for defining or highlighting features across the hair with a feature extraction process. Once these features are extracted, one can study the spatial statistics of these features, for example how uniformly are these features spread? How many of them are there? And what is the overall density and size of these features. These are the information obtainable from the sample without resolving it down to submicron level ( Fig. 4(b)). In other words, the method sacrifices localized resolution for obtaining larger scale image to capture the overall statistics of the features. To the best of our knowledge, there is no study to show how exactly the statistics of the scattering spots on the hair surface should change with regards to the hair damage level. This study is the first attempt toward this end, our initial investigations show that denser and uniformly distributed scattering spots are strongly correlated with higher damage level. Based on this initial observation, ideally to quantify this correlation, we needed to develop a metric that: a) responds linearly to the damage level of the hair, to avoid amplifying higher damage levels or neglecting lower ones; b) captures the density and distribution of the features while responding minimally to the global shifts of the features in the region of interest (ROI) (e.g. because of the cylindrical symmetry of the hair there should be no difference if the spots are mostly on the perimeter or the center, and c) can work for all types of hair with minor or no parametric initialization. The region of interest (ROI) is extracted by measuring the largest region (largest number of pixels in extracted region) from binary (black and white) image where there are detectable features. The red color indicates extracted region as shown in Fig. 4(c).    Fig. 4(c). This region is used for defining the boundary of feature extraction. If we didn't set such a boundary, then erroneous features from unwanted areas would enter the evaluation. Figure 4(e) shows the cuticle peeling features, which are extracted by using extended-maxima transform [25]. This transform finds regional maxima with 8 connected components (8 neighboring pixels). The centroid of each extracted blob I R (x,y) is used for final features to calculate feature distribution quality. To find regional maxima, we set the threshold value of the extended maximum transform as 15 for all of the images. This number which is a threshold on the pixel intensity is obtained heuristically and can be affected by the contrast and noise level of images in the system. If this value is set to 30, for instance, many of the features will be ignored and if it is set to 2 some of the noisy pixels will also be detected as features. The dimension of this window can vary depending on the magnification of the objective lens. The spatial and statistical distribution of scattered features are studied by using Delaunay triangles (DT) [26], histogram density (HD), and Voronoi diagram [26], [27]. These metrics are commonly used for image analysis and any other application that deals with computing in Euclidean geometry [28].

A. DELAUNAY TRIANGLE (DT)
A set of Delaunay triangles (DT) is computed for each input image by using extracted features as shown in Fig. 5(a) (Input image).
The area and the shape of each triangle can be used to represent the distribution of points [29], [30]. The area DT A and radiance DT R of Delaunay triangles are used to calculate distribution quality of hair as in Eq. 1 to Eq. 4. where n is the number of Delaunay triangles, A i is the area of the i th triangle, and A represents the mean area of the triangles. Also to calculate DT R equation below is used: where R i indicates maximum radian value of the largest angle among three angles (θ 1 , θ 2 , θ 3 ) of the i th triangle computed as [26]. (3) Higher DT value indicates a greater spread of features. We used DT based on simplicity and computational cost for portable applications. This is not necessarily the optimized approach. The bin size was fixed to 20 × 20 for all the measurements. Certainly, all our parameters are dependent on this bin size; for example, an extremely large bin size will saturate the HD parameter to 1 and a too small bin will push HD toward zero. The number 20 × 20 is chosen to represent and average feature size with slight margin around it. This number could be changed for different objectives with different point spread functions and image sensor sizes. The optimal number for the bin size should represent an average feature size on the hair which is about few microns (2-5 µm). The total probability indicates the overall spreading of the features. The higher probability indicates more widespread of damage.

C. VORONOI DIAGRAM (VD)
The mean value of Voronoi tessellation area of a feature set is calculated using Voronoi diagram (VD) (Fig. 7). The boundary of Voronoi diagram is computed using convex hull of feature set ( Fig. 7(b)) [31]. This boundary limits the area of the cells at the margin of the diagram to enable measuring the average area that contains a feature. The VD value indicates average size area that contains a feature. A very small VD indicates that the damages are widely and densely spread across the hair surface; larger VD can indicate that features are just localized in one area.

D. DAMAGE LEVEL CALCULATION
Higher values of DT and HD depict a higher damage level. To classify hair condition, these two parameters are simply multiplied to represent the damage level (DV) as We can observe that there is a notable meaningful difference between the calculated damage value for ''healthy" and ''damaged" hair in further experimental results with the dataset.

A. SIMULATION RESULTS
To demonstrate the operation of the proposed algorithm, and investigate the dynamics of its response, we have generated a synthetic data set as shown in Fig. 8. Figure 8 indicates that the distribution quality correlates with DT and VD super linearly while it correlates with HD sub-linearly. In our study the contrast between the damage values of samples are large enough to justify simple thresholding in the decision-making block of Fig. 3. Inclusion of DT and VD parameter not only brings the damage value metric to a more linear metric, but DT and VD will also work together to make the metric more robust to the random global shifts of the features in ROI. More complex decisionmaking algorithms are conceivable for further sophistication. For example, if the damage is a combination of different types of damage more complex output can be considered for the decision-making block. Here synthetic data set was used for fixed ROI size (Fig. 8: 50 × 50). In reality, the ROI size may differ from every acquisition of different hair sample. We observed the proposed method provide damage value in term of the varying ROI size as table 1.

B. EXPERIMENTAL RESULTS
Our samples were composed of three natural hair colors: black, blonde, and grey (or white due to aging). Within each sample set we had four categories of hair: healthy,  Measurements at three spots on a single hair for healthy and UV-exposed hair for three different hair colors. The algorithm successfully finds the damaged hair in all the cases. discolored (bleached), ironed, and 96 hours UV-exposed hair. While other types of damage are known to explicitly express mechanical changes in the human hair, UV exposure damage is known to be very subtle [32]. In fact, in earlier years it has been controversial between different researchers as to if UV damage express itself in the physical structure of the hair [9]. As noticed in Fig. 9 there are clear difference in the number and density of the scattering clusters in healthy and UV exposed hair. It is noteworthy that the imaged features are sparse and also not fully resolved as seen in Fig. 9. However, due to sparsity we can yet study the statistics of these features and, therefore, categorize the hair samples as in Fig. 10(a). Fig. 10(a) shows the damage level for different hair. By merging data from all hair samples we can observe the proposed method finds the damaged hair in all cases using a certain constant threshold. This threshold is calculated by k-means clustering of all data sets. It is clear that the damage value is strongly correlating with the damage level of the hair. Fig. 10(a) further shows that not only the introduced metric can categorize between healthy and unhealthy with 100 percent accuracy in our sample set, but it can also categorize between different types of damage on average. This is because all the averaged curves are in parallel and well distanced from one another. The ground truth is based on the preparation process of the hair samples in the laboratory. For example, the UV exposed hair is exposed to UV for 96 hours with UV radiance mimicking the sun profile at sea level from 280nm to 400nm whereas the healthy hair is not. Since we are studying a biological sample and not a physical sample, it is difficult to pinpoint a sharp gold benchmark on what is a ''healthy'' hair; as hair chemical structure and geometrical features varies by race and from person to person. However, this does not mean that one cannot measure the impact of external damaging factors on biological systems. Here when we refer to healthy hair, we are referring to an un-impacted hair or part of the sample batch that hasn't been exposed to the specific damage factor. The hair samples are collected from healthy adults and elderly with no medical condition or malnutrition by Natura Company under due clinical standards. In general, the results show that the damage value is about 3 to 4 folds higher for UV-exposed hair compared to healthy hair. This value is about 5 to 6 times higher for bleached hair and finally about 7 to 8 times larger for flattened or ironed hair. This can provide an expected damage value range that can categorize the hair quality. For example, any hair sample that has damage level above 0.3 can be considered heavily damaged most likely because of flattening. As Fig. 10(a) indicates, there are some differences in these values for different hair colors. Grey hair has the largest error bars. In general, the standard deviation for DV is about 0.02 for un-impacted or healthy hair samples whereas the standard deviation of this metric increases to 0.1 for impacted or damaged hair. The standard deviation as indicated in Fig. 10(a) for the measurements is the highest for damaged gray hair (specifically flattened gray hair) and lowest for dark brown healthy hair. This is expected as gray hair already has anomalies on its structure due to aging process and it is further refractive to the visible range light which create further noise in the measurements. This is completely the opposite case for healthy dark brown hair where the hair itself is very well absorptive of the spectrum. This strong absorption assists the clearing of the background scattering and results in a better contrast for the scattering spots. The values do however show a consistent trend with regards to different types of damage as shown in Fig. 10(a). While in some cosmetic applications a rough damage value may suffice, further extended measurements are necessary for a more demanding clinical application with regards to damage type. This may require some automation on the hardware to reduce the data acquisition time. It must be noted that the damage level is relative and not absolute. The level can change from setup to setup and therefore there can be some relative shifts in the values. For example, we disassembled and realigned the setup for Fig. 10(b) which shows the results for three measurements at different parts of single hair for all three colors. As noted, the variation for the same hair is much smaller. Table 2 shows an example detailed data set for one set of measurements. One of the factors that can contribute to this damage level change in these two setups is the change in image size on the sensor and not updating the relative bin size accordingly. Another factor is the slight changes in the tilt of the sample holder and alignment of the optics, e.g. change in aperture size or angle of polarizer. Finally, the last factor that can contribute to the shift in the parameter is the background light that can vary from setup to setup due to slight variations in illumination geometry, camera acquisition parameters, and optical isolation of the setup from surrounding light.

V. DISCUSSION
While our study shows that lower magnifications can be used for categorizing different levels of hair damage, at this point we were unable to resolve the damage features properly with lower end 10× (0.25 NA) objectives. However, it is conceivable that if more than one strand of hair is used simultaneously, or if higher resolution sensors (as those in modern smart phones) are used there can be a better SNR for feature extraction. It is noteworthy that while our algorithm demonstrates the successful computational human hair categorization, it is by no mean the optimized or the most accurate algorithm; therefore, finding better and more accurate algorithms to match even lower magnification objectives can be the topic of future research. The technique can be used for other colors and potentially artificially dyed hair, however, the colors chosen here make up majority of the color composition of the population. There is some variance in the data that changes based on the hair color and type of the damage. In real-time applications the DV (damage value) can be calculated in per frame however as in any other realtime processes there can be a tradeoff between frame rate and accuracy of the results. While the hot spots themselves can be notably smaller than the resolution of the optical system that we are using; this does not mean that our measurement is not impacted by noise. The impact of noise in our detection process is not linear as there are multiple thresholding processes that affect the data. The noise initially impacts the process through feature extraction step that is performed via extended-maxima transform. This error then propagates to the rest of the process and impacts the extracted statistics. The effect of noise and background illumination is more evident in grey hair samples as the background illumination is closer to the illumination level of hot spots created by peeling cuticles.

VI. CONCLUSION
By computationally exploiting the sparsity of mechanical features on human hair we enable categorization of hair quality and damage type under low magnifications (as low as 20×). Our use of polarized dark-field scattering and Euclidian metrics such as Voronoi diagrams and Delaunay triangles provides a single numeric metric that shows significant contrast (4 to 8 fold contrast) between different types of damage. This technique is not optically or computationally expensive and, therefore, is highly appealing for portable form factors and real-time applications both in cosmetic and clinical applications.
[30] Q. Zhu KRISHNA RASTOGI received the B.E. degree in biotechnology from the PES Institute of technology (currently PES University), in 2015. His research interest includes inventing low-cost medical device for mass screening. He is currently the Founder of a start-up, currently working on distributed learning on decentralized data, which provides customized hardware and federated AI solutions without compromising on the privacy of the client data.
RAMESH RASKAR joined the Media Lab, Mitsubishi Electric Research Laboratories, in 2008, and the Head of the Lab's Camera Culture Research Group. He is currently an Associate Professor with MIT Media Lab. He holds over 80 U.S. patents. He is also coauthoring a book on computational photography. Recent projects and inventions include transient imaging to look around a corner, a next-generation CATScan machine, imperceptible markers for motion capture (Prakash), longdistance barcodes (Bokode), touch+hover 3D interaction displays (BiDi screen), low-cost eye care devices (Netra, Catra), new theoretical models to augment light fields (ALF) to represent wave phenomena, and algebraic rank constraints for 3D displays (HR3D). His research interests include the fields of computational photography, inverse problems in imaging, and humancomputer interaction. He received the TR100 Award from the Technology Review, which recognizes top young innovators under the age