A Hybrid Algorithm for Face Detection to Avoid Racial Inequity Due to Dark Skin

There has been significant development in the facial recognition technology during past few decades. This technology has been widely used by different organizations and governments for defense, security, and surveillance projects. Furthermore, it has now been incorporated into our daily usages, such as consumer applications, personal data protection, or cyber-security, particularly while using smartphones. Most of these systems work very efficient, however, there are some challenges related to the accuracy of results of facial recognition systems when tested on images of people with dark skin. As a matter of fact, various studies demonstrate higher accuracy when tested on data set with white skin personnel, while exhibit a much lesser accuracy when the same algorithms are tested on dataset of people with dark skin. This article highlights the variation in accuracy of existing facial recognition algorithms when applied to dark-skinned people. Furthermore, as a principal contribution it presents a hybrid algorithm based on Gaussian and Explicit rule model that improves the accuracy for face-detection for dark skinned people. Thorough experimental evaluation has been conducted with a data set of black faces by first identifying skin and non-skin regions and then applying skin segmentation. The results have been compared with existing face detection algorithms with a clear improvement in the accuracy of 89% for dark skin.


I. INTRODUCTION
Face detection is a biometric system utilized for person identification from any digital image by comparing the person's facial features with all of the images already stored in the system database. There has been much research conducted on facial identification and widespread application in security and access control management [1]. Face recognition has also been applied in law enforcement surveillance, passenger screening at the airports, security background checks for hiring employees, and banks, especially when approving loans. Considering the application of daily use, now we can very easily unlock our iPhone with a glance or tag people on social media sites, e.g. Facebook [2]. However, the human face is vulnerable to multiple variations like age, light effects, and expressions of the face, while capturing images and image quality/resolution. These variations act as the challenge for face recognition techniques to yield accurate results [3]. Several research efforts have shown that skin color The associate editor coordinating the review of this manuscript and approving it for publication was Eunil Park . variation in the facial recognition mostly acquires inaccurate results [4].
Multiple approaches has been developed to improve face detection accuracy, especially to deal with the variations that affect facial recognition results [34]. Lowe's Scale Invariant Feature Transform (SIFT) was one such initial technique. The SIFT technique was also considered as one of the most constant feature descriptors. Initially, SIFT was used in object detection; however, later, it became the base of system vision application's algorithms. This method starts by initially transforming images to the local feature vector by identifying certain key points and then applying quantitative information to furnish the results for object recognition. Vectors of the local feature remain constant for translation of images, rotation, scaling, and illuminating changes without compromising accuracy. However, SIFT descriptors were only designed to work for gray images [34].
In facial recognition, color acts as a major variant. Classification of human skin color helps to identify a person's skin tone achieved by definition of skin region. This technique has simplified detection rules and helps in generating much faster VOLUME 9, 2021 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ FIGURE 1. The gender shades project discrepancies [3].
classifiers [8]. There are other skin modeling techniques, such as neural networks [12], Bayesian networks [14] and k-means clustering [13]. However, skin modeling has been complex and challenging task and effected due to illuminating conditions In addition, face attributes like occlusions, pose, illumination, expressions need to be reduced and normalized to achieve further accuracy in face detection. Transfer learning approach and clustering techniques can be applied to cater this challenge [30]. The variation among the skin tone within races has been considered as major challenge for all skin modeling techniques [20].
Most of the face recognition algorithms claim to have high classification accuracy on some datasets, however these results were not the same when verified among different demographic groups. The most inaccurate results were found in the black female subjects aged 18-30 years [5]. The Gender Shades project was conducted to evaluate the accuracy of Artificial Intelligence based Gender classification applications [33]. Three gender classification products have been chosen that were developed by tech companies -Microsoft, IBM, and Face++ for evaluation of results. In this research, the subjects were grouped into four categories: females with dark skin, males with dark skin, females with light skin, and males with light skin. All the algorithms showed the worst results for females with dark skin having errors up to 34% higher than the people with lighter skin [6].
Another independent research was conducted by the National Institute of Standards and Technology that depicted somewhat similar results as shown in Figure 1. Around 189 algorithms were verified, and they were the least accurate for females with dark skin [3].
The novelty of this paper is to present a hybrid algorithm using the Gaussian Model and Explicit Rule Algorithm for skin detection systems that work effectively for both light skin and dark skin people. Face detection systems have now been used commercially, however, the issues to detect people with dark skin have raised doubts regarding the accuracy of the results of these systems. This research is not limited to face detection systems/applications only but can also be applied in other research domains such as classification of malignant melanoma skin cancer on people with dark skin [31]. The CNN based classifiers have been applied to detect skin cancer effectively; however, detecting skin lesions on people with dark skin is still a problem. Gaussian and Explicit Rule face detection algorithm can be applied in these research domains to enhance accuracy results for darker and lighter skin.
The subsequent sections of this paper are categorized as follows; following the introduction section, related works are presented. Section II discusses the issues in the existing face detection algorithm and the accuracy rates when tested with a different data set based on regional diversity. Section III presents the hybrid algorithm that improves the accuracy of the face detection system when tested with black skin. In conclusion, we made a comparison of the previous and the proposed algorithm.

II. RELATED WORKS
This section explains the research efforts that address issues due to skin color and the related work done in the same domain to improve the accuracy of face detection systems for people with dark skin. Across the globe, big technology companies provide facial recognition systems for commercial use like schools, airports, transports, city councils, shopping centers that were vulnerable to skin color [1]. Another research was conducted by Ms. Buolamwini [22] regarding the performance of facial recognition systems in the classification of images based on gender and race. She picked three major companies providing face detection systems, i.e., IBM, Microsoft and Megvii. Her research data set was varied from six different countries, comprising of 1270 digital images of parliamentarians. The results generated from the three systems, Microsoft products shows the best results with 94% accuracy. The other outcomes of her research were that all three products produced better results on male data set than females and on the lighter skin tones compared to the darkest skin tones. On the other hand, the worst results were shown for females with dark skin, with Microsoft shows an error rate of 20.8%, IBM 34.7% and Megvii 34.5% as show in Figure 2. The results show that the commercial face detection systems have failed for one of three dark skin women images and lack racial diversity of training images in the face detection system [3].
Another similar research was done by five different scientists based in the USA to understand the accuracy of face recognition systems based on gender, race and age of the data set of around 10,00,000 digital images. Unfortunately, this research also yields the same results and the outputs were worse for people with dark skin. Although this research depicted that accuracy could be improved by adding racial diversity in training datasets, recognizing the dark skin cohort could still be an unresolved issue [3].
An illumination invariant skin detection algorithm was developed [7], a hybrid skin segmentation system. Then another enhancement was presented [32] where the images in Red, Green, Blue (i.e., RGB) format were converted into YCbCr by making use of a combination of skin segmentation algorithm and Gaussian threshold distribution mode applied for the segmentation of skin [15]. However, when this hybrid algorithm was verified on faces with dark skin data, the detection accuracy rate reduced to 66.18%. The same hybrid method was applied to the combination of data sets like light, white and dark skin, the accuracy rate was improved to 92.35%. The results varied when the same method or algorithm of the face detection system was applied to different data sets based on racial diversity. Therefore, there was an immense need to work extensively on dark skin in face recognition systems. Another research was done on the Indian sub-continent race and a 91.1% accuracy rate was observed using the HSV model (Hue, Saturation, Value) [16].
The YCbCr color space heuristic threshold was used to detect skin tone. Explicit rules were implemented with respective thresholds to improve the detection of skin tone in color space. The outcome was that the image's pixels were classified as skin and non-skin. The detection rate was a high quality of 93%, with very few false positives observed [17], [18]. A combination of RGB, YCbCr and HSV was used to enhance the pixel identification. This combination algorithm used a different approach for segregating pixels of skin and non-skin regions. For each parameter, explicit rules were defined by a range of threshold values and combined bitwise using AND operation. The results of this research were also improved, yielding an accuracy of 94.43% [19], [32].

III. MATERIAL AND METHODS
Due to issues in face recognition systems to detect faces with dark skin, there has been a need to present a method that shall provide optimal results when identifying dark skin faces. This section presents a hybrid algorithm for detecting faces with dark skin using the Gaussian Model and Explicit Rule Algorithm. In this hybrid algorithm, a combination of morphological and anthropological methods has been applied for face detection. Furthermore, eye blinking and face corpora have been utilized to conduct the verification of the hybrid system. The results show that 87% of accuracy was achieved when skin detection was done using the Gaussian model. On the other hand, an accuracy of 71% was obtained with Explicit-rule. However, when the Gaussian and Explicit rule hybrid was applied, the accuracy improved to 89%.
An experiment was conducted using CCTV (Closed Circuit Television) cameras to capture digital images from video streams. These digital images were transferred for preprocessing of segments. Skin color has been the primary component for skin recognition in an image. However, there could be variations in color due to skin tone and racial diversity. Light also acted as an important variant that affected results. The image was broken down into discrete pixels to classify as skin and non-skin colored areas. A threshold-based combination of RGB and YCbCr values algorithm has been applied. Following factors have been considered to identify the threshold range: • Illumination impact on surroundings • Facial characteristics like age and gender • Background colors, blurriness, and shadows Ultimately, skin detection/projection methods were applied to face detection and segmentation.

A. DETECTION OF SKIN
The skin detection process started by collecting skin samples. Approximately three hundred digital images were gathered for training. Then estimation of mean, variance, and covariance of the training data set was done. Next, the Gaussian model was applied to model human skin to distinguish the non-skin pixels. Furthermore, a greyscale image was generated that depicted the possible skin regions. The next step was converting a greyscale image into a binary skin map to segregate further the skin and non-skin areas with the application of threshold methodology.
Gaussian density function defines the skin pixel color distribution with the probability of skin pixels represented by equation (1).
In this equation, Modeling of human skin was done using explicit rules from 2-dimensional and 3-dimensional color spaces. These rules define a threshold where the image pixels were segregated between the skin and non-skin areas. This approach has already been classified as the most effective approach for the classification of skin tone. The CbCR digital images were applied to the explicit-rule skin segmentation technique VOLUME 9, 2021 represented in equation (2) to highlight the skin area for black skin color tones: The output of both algorithms, i.e., the Gaussian model and the Explicit rule, were combined using the AND operator for getting the results of skin region through equation (3).

B. DETECTION OF EYE AND FACE
As mentioned in the above section, the RGB images were transformed into greyscale. For binary conversion, Canny Edge Detector was used. The eyes location was used to normalize the pose of the image. The eye pupil center was located using horizontal and vertical projections using equation (4).
x 0 , y 0 Coordinates were used for face alignment in equation (4). First, the line that joins the two eyes was made horizontal by angle θ rotation of face image, defined as the angle of elevation represented in equation (5). This process has been known as de-skewing and improves the effectiveness of the algorithm.
(x, y) = Left Eye Coordinates x 1 , y 1 = Right Eye Coordinates The inner distance between the eyes was used for face normalization as in Figure 3. 'd' defines the distance among both eyes. This point had been regarded as the center for making segmentation and rounding the face and the eye region.

IV. EXPERIMENT AND RESULTS
A data set of 300 images was used to get dark skin samples and application of Gaussian probability distribution. Figure 4 depicts the results of Gaussian probability.
• Figure 4(a) represents the real image used as the data set. • Figure 4(b) highlights possible skin areas after the application of the Gaussian method.
• Figure 4(c) depicts skin region segmentation after application of threshold on Gaussian probability distribution. The white portions of the images have been referred to as skin regions as shown in Figure 4(c), and the black portions are non-skin regions. The 79 out of 90 videos from the given data set have been detected people with dark skin, and it yields the detection accuracy of 87%. However, 17% have been detected as false-positive.
For mapping segments of the dark skin, explicit-rule skin segmentation model was used. Dark skin samples of around  2700 pixels were estimated from the 300 images data set. Cb and Cr color space histogram was used for chromatic color space distribution. Dark skin thresholds were identified, where Cr the lower limit was 85, whereas the Cr upper limit was 200 (85 < Cr < 200). Similarly, for Cb the lower limit was 60 while the upper limit was 130 (60 < Cb < 130). The histogram valley method has been used for the selection of threshold values. Any of the pixels that fall within this threshold boundary have been considered as dark skin pixels. The multiple model point compared with light skin due to variety of tones in people with dark skin. Explicit-rule skin map enhanced accuracy for identification of skin and non-skin classification and the detection accuracy of 71%.
Gaussian probability has been combined with Explicit rule skin mapping using AND or OR operator to provide more accurate results. The figure 5 shows the hybrid skin map samples where 89% of AND hybrid shows accuracy where 55% accuracy has been observed with OR hybrid.
Dark skin image from the hybrid skin map was then mapped to the original RGB digital image for skin segmentation, as shown in Figure 6 for face segment processing. The figure 7 depicts the skin segmentation done over the images.
The Table 1 represents a comparison of data preprocessing results with existing algorithms. The results showed an increase in the precision rate when the proposed model had been applied for data preprocessing.
The Table 2 represents the results of skin detection when Gaussian and Explicit Rule Hybrid algorithm has been applied to the dark skin data set. The results depicted the increase in detection rate of 89% when the Hybrid Model was used for skin detection of people with dark skin.
The Table3 shows the comparison of results with existing solutions for the detection of people with dark skin.

V. CONCLUSION
This paper has highlighted the issues in existing face detection systems for people with dark skin. This research emphasizes the improvement in percentage inaccuracies of existing commercial face recognition systems and the need to improve existing algorithms to deal with racial diversity. However, different results were produced when the same algorithms were verified with another data set based on skin color and racial diversity. Therefore, enhancing the training data set have not suffice to avoid the issues in face detection of black faces. This paper introduces a hybrid Face Recognition Algorithm that combines Gaussian Probability and Explicit Rule algorithm. The results showed that Gaussian and Explicit Rule hybrid algorithm optimally improved the face detection rate for people with dark skin. A comparison between the existing and the proposed algorithms has been made based on the accuracy rate in skin detection and data preprocessing. A detection rate of 89% has been achieved when working with dark skin, which improves the current 83.3% detection rate. In the phase of experiment, preparation of data set was crucial because fewer dark skin people were willing to participate in the experiment. In the future, the research shall be conducted on a large scale data set with a higher percentage of racial diversity within people with dark skin to strengthen the results. The proposed hybrid algorithm can also be used in other research domains, such as medical research related to skin related diseases of people with dark skin.