Automating the ABCD Rule for Melanoma Detection: A Survey

The ABCD rule is a simple framework that physicians, novice dermatologists and non-physicians can use to learn about the features of melanoma in its early curable stage, enhancing thereby the early detection of melanoma. Since the interpretation of the ABCD rule traits is subjective, different solutions have been proposed in literature to tackle such subjectivity and provide objective evaluations to the different traits. This paper reviews the main contributions in literature towards automating asymmetry, border irregularity, color variegation and diameter, where the different methods involved have been highlighted. This survey could serve as an essential reference for researchers interested in automating the ABCD rule.


I. INTRODUCTION
The incidence of skin cancer is rapidly increasing throughout the world and is becoming one of the deadliest forms of cancers, especially in countries with large Caucasian population [1], [2]. It is considered the most common cancer worldwide such that one in every three cancers diagnosed is skin cancer, in which Malignant melanoma is the third most frequent type [3]. Although melanoma constitutes only 1% of the diagnosed skin cancer cases, it causes 75% of deaths [4]. This increasing incidence of melanoma deems the attempt of the early detection of melanoma a continuing public health priority. Despite its aggressive infiltration of other body parts, melanoma is highly curable if diagnosed early and treated timely [5], [6]. Early detection is crucial since it contributes to better survival: the 5-year survival rate for early stage invasive melanoma is 94%, compared to a 5-year survival rate of only 17% for melanomas that have spread to other parts of the body. There is also a significant cost decrement when melanoma is diagnosed at an earlier stage. There is a niche on developing an objective, bedside tool that could be used as an adjunct in the clinical assessment of skin lesions. Manually tracking tumor changes is also labour-intensive, especially for patients with multiple moles on their skin.
Dermoscopy is an in-vivo, non-invasive technique that uses different incident light magnification systems with an oil The associate editor coordinating the review of this manuscript and approving it for publication was Pengcheng Liu .
immersion technique [7], [8]. It provides dermatologists with a technique for inspection of skin lesions, rendering higher accuracy for detecting suspicious cases that would not be possible via inspecting with the naked eye. Depending on the observer's experience, dermoscopy improves the diagnostic accuracy for melanoma up to 50% as compared to purely visual inspection. However, dermascopic diagnosis is subjective and has poor reproducibility and low accuracy especially for inexperienced dermatologists. The accuracy of experts is in the range of 75% -84% [9], [10]. Diagnostic accuracy can also be affected due to fatigue, especially with a large number of images the dermatologist has to interpret in a limited amount of time. A report from the National Cancer Intelligence Network in 2016 found that GPs failed to refer almost one in three patients with malignant melanoma for urgent tests.
The ABCD rule (criterion) [13] -which is an acronym that refers to the features: Asymmetry, Border irregularity, Color variegation, and Diameter greater than 6mm -emerged in 1985 by a group of researchers at the New York University as a simple framework that physicians, novice dermatologists and non-physicians can use to learn about the features of melanoma in its early curable stage, enhancing the early detection of melanoma. It is more geared towards the public than the 7-point checklist which was designed for nondermatological medical personnel. The approach was then verified by the 1992 National Institutes of Health Consensus Conference Report on the detection and treatment of VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ melanoma, in addition to other studies published at the time [14]- [17], and is being advertised by the American Cancer Society as a method to help the early medical evaluation of any suspicious pigmented lesions. The ABCD features provide simple means for appraisal of pigmented cutaneous lesions that may need to be further examined by a specialist, which might result in further work of dermoscopy or biopsy, or both. The rule is basically designed to be used on a daily life basis by both the layperson and the primary care physician (non-dermatologist) as a simple method to alert on the clinical features of melanoma, and is intended to help explain a subset of melanomas called thin tumors, which could otherwise be confused with benign pigmented lesions. It should be noted that the combination of the ABCD features (e.g., AB, AC, ABC) is what determines the suspicious lesions and has a greater accuracy when used in combination, especially when it is not necessary for all melanomas to acquire all the four features. Referring back to the results of the studies that attempt to document the diagnostic accuracy of the ABCD criterion in clinical practice, combining the reliable sensitivity, specificity and adequate interobserver concordance in the application of the ABCD criterion supports the ongoing usage of this criterion in clinical practice [18]; the easy-toremember ABCD criterion is used in public education on a broad basis [19].
There have been significant advances of fully and semiautomated computer aided diagnosis (CAD) methods in recent years [11]. These CAD methods can assist dermatologists in different steps of analysis such as the detection of lesion boundary, quantification of diagnostic features, classification into different lesion types, and visualization [12]. In fact, CAD systems are able to obtain a higher level of sensitivity for melanoma detection compared with inexperienced dermatologists [106]. The standard pipeline in automatic dermoscopic image analysis consists of three main stages: image segmentation, feature extraction/selection, and lesion classification. There have also been numerous efforts to automate the ABCD rule in an attempt to provide objective approaches to finding the different features the rule represents. In this paper we review the major publications written on the topic and demonstrate the different approaches used to automate the ABCD traits. We believe that this work will serve as an essential reference for researchers on the subject, especially that to the best of our knowledge no similar review has been published in literature before.
Sections II, III, IV and V highlight the approaches geared towards automating asymmetry, border irregularity, color variegation and diameter, respectively. Comments and discussion on the reviewed methods are provided in Section VI, and conclusions are drawn in Section VII.

II. ASYMMETRY
(Asymmetry refers to the fact that when drawing a line across the middle of the mole (Fig.1), the two halves will not match, that is, the shape of one half does not match the other half, providing a warning sign of melanoma. Asymmetry evaluation is carried out by separating the lesion into four sectors using orthogonal axes that pass through the lesion centroid and are aligned so that minimum asymmetry (maximum symmetry) is obtained [20]. Different attempts have been made to automatically determine the asymmetry of skin lesions in literature, among which Circularity ( 4πA P 2 ; A: area, P: perimeter) is the main measure used (as in [50]) .
Ng and Cheung [21] used Symmetry Distance (SD), which is the minimum distance required to move points from the shape P to the symmetry transform ST (P) -a transformation that converts P into a symmetric shape closest to it. However, this approach becomes computationally expensive when there are many points on the skin lesion border. Even if selecting fewer points, this selection will impact the SD value and is dependent on the real shape of the lesion to be measured. The authors proposed an algorithm for combating the issue. To take the irregular borders into account and how they might affect the segmentation output, a Fuzzy Symmetric Distance (FSD) is proposed to improve the discriminative power of the SDs by widening its numeric range for the lesion images. FSD provides the best performance when compared to SD and circularity. Vincent et al. also used SD in their work [22].
Stoecker et al. [23] proposed a method where the x and y coordinates of the image were made to coincide with the centroid of the image through shifting and translating the shape, the x and y coordinates were then aligned with the centroidal principle axes by rotating the image. The image is eventually reflected across the principal axis and its orthogonal counterpart. Two area differences are produced by subtracting the image on one side of the axis from the reflected image, where the least of the absolute values of the area differences A min is divided by the skin lesion area A, resulting in the asymmetry index = A min A × 100% (a similar approach was used in [48], [50], and [64]). A classification step is then carried out by finding the best asymmetry index threshold, such that skin lesions having an index value greater than a threshold are classified as asymmetric. A 6% area difference is used as the asymmetry threshold to achieve around 93% accuracy. 83334 VOLUME 8, 2020 Seidenari et al. [24] measured symmetry by dividing the skin lesion into 256 sectors using axes that were passed through the barycentre of the lesion. The difference in symmetric areas with respect to the center is then found, evaluating to symmetry values that ranged from 0 (symmetric) to 10 (asymmetric). The symmetry measure is used as part of a discriminant analysis classifier that combines other variables (measures) with different weights; a threshold is derived for lesion classification. Andreassi et al. [25] employed a similar approach where the area differences were taken between 360 lesion segments. d Amico et al. [26] found asymmetry by splitting the skin lesion into two halves via a straight line that passes through the center of mass. Comparison of the two halves is then made by computing the distance between their size functions [27], which are maps from the plane to the natural numbers. Size functions have two inputs: object (lesion boundary) and a real map (measuring function, such as the distance from the center of mass). The splitting is repeated for 45 equally spaced radial lines, resulting in the distance as a function of angle. A set of characteristic numbers are extracted from the generated curve that are eventually fed to a Support Vector Machine (SVM) for classification. The advantage of this approach lies in detecting qualitative asymmetry as opposed to the traditional methods proposed which focus on geometrical asymmetry.
Ma et al. [28] firstly defined the major and minor axes that partition the skin lesion into four parts, and the asymmetry measure of the contour is found by calculating the normalized asymmetry degrees k of a pair of contour segments with respect to the central, major and minor symmetry. Features (namely the area, averaged distance of local fractals, relative radial distance and normalized zero-cross rate) for each contour segment that would determine the degrees of asymmetry are then extracted. When k approaches zero this means that the lesion is symmetric. Results show that the extracted features have a high capability of discriminating between moles and melanomas.
The integration of Fourier descriptors into a shape asymmetry quantifier was proposed in [29], where the best symmetry axis is determined based on the proximity to zero of the real parts (coefficients) of Fourier descriptors. The sum of those descriptors measures the symmetry such that the minimum value of the absolute sum of real coefficients denotes the minimum asymmetry. It has been shown that such descriptors could aid in identifying the skin lesion's principal axes of symmetry accurately, as when using the top two unique axes determined by the proposed approach a 92% match with the dermatologist is achieved.
After segmenting the skin lesion, Tenenhaus et al. [44] considered both shape and texture asymmetry. In shape asymmetry, two symmetry axes calculated with the Hough transform algorithm are formed and asymmetry is estimated for each axis from the percentage of overlapping pixels after rotating the tumor surface around the axis. Texture asymmetry is estimated by finding the quadratic error averages between the intensity of overlapping pixels after rotating the tumor around the symmetry axes. Asymmetry is eventually described by a 4-dimensional vector, and this feature vector is fed to a KL-PLS (Kernel Logistic Partial Least Square regression) based classifier [61] as input which is then used to detect asymmetry. The approach is tested on 227 tumor images resulting in 73% accuracy.
She et al. [110] measured asymmetry by folding the lesion outline (i.e. segmented image) about the major axis of the best-fit ellipse, finding the non-overlapping region, and calculating the percentage of the non-overlapping region over the lesion area using the equation A = T T × 100%, where T is the number of pixels in the non-overlapping region, and T is the lesion area; A is called bi-fold method [51].
After the skin image is converted to grayscale and segmented, Zaqout [51] partition the skin lesion (foreground of segmented image) across its centroid into two equal halves both vertically (left vs. right) and horizontally (top vs. bottom). To measure asymmetry, the entropy between each corresponding half is found and the entropy asymmetry is calculated based on a specified entropy threshold value. The bi-fold method is also applied and the overlapping asymmetry is calculated based on an overlapping threshold. The average value of those asymmetry results represent the overall asymmetry score of the segmented lesion.
Kasmi and Mokrani [46] first determine the axes of symmetry (horizontal and vertical) then measure color asymmetry, brightness asymmetry, and shape asymmetry. In color asymmetry, the image is divided into blocks of 20×20 pixels, such that blocks with at least 75% of their area inside the lesion are selected. The image is converted to the L * a * b * color space, the color average of each selected block is computed, and the color distance (i.e. Euclidean distance) between two symmetrical blocks along the principal axis is measured. The authors set a JND (just noticeable difference) as a threshold for differentiating between colors, such that along any axis for any pair of blocks if JND is smaller than the specified threshold the block pair is said to be color symmetric. If along any axis the number of blocks that are color symmetric is greater than the number of blocks that are color asymmetric, the image is considered to be color symmetric along that axis. Brightness asymmetry is measured by finding the difference between the average luminance of the two opposite halves of the skin lesion along a symmetry axis. If the difference is less than a specified threshold (3% of the total average luminance), the lesion is considered to be brightness symmetric along that axis. Shape asymmetry is measured by finding the difference between the lesion areas of the two opposite halves and comparing it with a specified threshold (2% of the lesion area). A lesion which is simultaneously color, brightness, and shape symmetric along an axis is considered symmetric along that axis. If the lesion is symmetric along both axes, along only one axis, not symmetric along any axis, the asymmetry score is set to 0, 1, or 2, respectively.
Ali et al. [52] built a vector of three measures to find skin lesion asymmetry: SIFT based similarity, projection profiles VOLUME 8, 2020 and skewness. In particular, the authors split the extracted skin lesion vertically and horizontally across the center of the images into four halves, and used SIFT (Scale-invariant feature transform) [53] to measure the similarity between each opposite half. The total similarity score is measured as v s +h s , where v s and h s are the vertical similarity and horizontal similarity (matches/correspondences), respectively. Higher values of this measure indicate more similarity between the two halves. In the second measure, projection profiles, symmetry is found by projecting the segmented lesion in the x and y directions and then comparing their histograms. The horizontal and vertical projections represent the number of foreground (skin lesion) pixels in each row and column, respectively, where they are eventually represented as a histogram. The two histograms are compared using a correlation method, such that the higher the correlation value, the more symmetrical the skin lesion. The third measure, skewness, is finally found to represent a distribution's (skin lesion image) degree of deviation of the respective projection from symmetry, provided that skewness evaluates to zero if the projection is symmetric with respect to the origin (mean). The extracted asymmetry measures are used to train a decision tree, which is then used to predict the asymmetry of new skin lesions, achieving an 80% accuracy in determining asymmetry in skin lesion images.

III. BORDER IRREGULARITY
Unlike benign mole borders that tend to be smooth and even, early melanoma borders possess uneven (irregular) borders. Fig.2 shows samples of skin lesions with regular and irregular borders [54]. In [30], a dermatologist was asked to score 60 skin tumor images as being regular or irregular (regular: 14, irregular: 46). A border is then found using a radial search algorithm [31], where a flash threshold is first heuristically determined such that image points with luminance greater than the threshold are classified as flash reflections and not used in selecting the tumor boundary points. The user then decides the number of radial lines at which the program will use to select candidate border points, such that the tumor boundary points crossing along a radial line are identified by finding the first significant jump in average luminance that would be sustained for a specified length (i.e. number of consecutive pixels along the radial line that comply with the jump threshold). Irregularity is eventually found using this formula: I = P 2 4πA , where P and A denote the perimeter and area of the closed boundary, respectively. The perimeter is measured by counting the points on the detected border, and the area is measured by counting the points on and within the border. The authors concluded that borders with an irregularity greater than 1.8 were classified as irregular. Using the proposed algorithm, 42 of the 46 irregular tumors are classified correctly. Of the 14 regular tumors, 8 are classified correctly. Thus, 83.3% of the tumors are classified the same as the dermatologist. This irregularity formula (i.e. I) has also been used by She et al. [110] and Messadi et al. [64] (combined it with compactness, fractal dimension, and radial variance).
Ng and Lee [32] used fractal dimensions (FDs) in measuring the irregularity of skin lesion borders. For each color image, four fractal dimension measures were found: direct FD, vertical smoothing FD, horizontal smoothing FD and multi-fractal dimension of order two. Those FDs were also calculated on the blue band of the images. After being segmented by a multi-stage method [33], 468 melanocytic lesions (not hairy) are used to test the proposed approach. Results show that the multi-fractal method performs the best. Other work the uses FDs were used is found in [34] and [54].
An automatic approach for analyzing the structural irregularity of cutaneous melanocytic lesion borders was proposed in [35]. The algorithm consists of two stages. In the first preprocessing stage, the lesion border is extracted from the skin images after removing the dark thick hair by DullRazor [58]. In the second stage, the structural shape of the lesion border is analyzed using a measure called sigma-ratio, which is derived from the scale-space filtering technique with an extended scale-space image. Results show that unlike shape descriptors such as compactness index and fractal dimension that are more sensitive to texture irregularities than structure irregularities (i.e. don't provide accurate estimation for the structure irregularity) [36], sigma-ratio is considered sensitive to the structural indentations and protrusions. The author further improved their past work to propose a new border irregularity measure in [36]- [38]. The new method works first by locating all indentations and protrusions along the lesion border, and a new irregularity index is measured for each indentation and 83336 VOLUME 8, 2020 where I U is the irregularity index of an indentation/protrusion segment U , A U is the irregularity area for segment U , and R U is the area of the smoothed contour. Summing up all the individual indices provides an estimation on the overall border irregularity. A new feature is also introduced in the proposed method in that it is able to localize the significant indentations and protrusions.
Arbisala and Claridge [39] proposed a new measure of border irregularity based on conditional entropy, where it was observed that the entropy increases with the degree of irregularity. Ninety-eight skin lesions are used in the experiments, of which 16 are melanoma. The results of the proposed measure are compared with the Indentation Irregularity Index (the sum of irregularity indices for all structural indentation segments) [38] and show to have a better discriminatory power such that the area under the ROC curve is 0.76 compared to 0.73 for the Indentation Irregularity Index. In particular, the proposed measure gives 70% sensitivity and 84% specificity.
Ma et al. [40] used wavelet decomposition to extract the skin lesion border structure, based on which they would determine whether the lesion is a naevus or melanoma. Using the discrete wavelet transform (DWT), the 1D border (2D lesion contours are modeled as 1D signatures to be position invariant) is filtered into sub-bands down to level 9 where levels 6 to 9 (significant levels) have shown to contain information considered best for classifying between melanoma and benign samples. Some statistical and geometrical feature descriptors of border irregularity are extracted at each individual sub-band. A back-projection neural network is used as a classifier which receives a combination of features as input. Twent-five measurements are formed by applying 6 features in 4 significant sub-bands, and 1 feature in a single sub-band. Using a small training set of 9 melanomas and 9 naevi, the best classifier is obtained when the best 13 features are used.
A system was proposed by Jaworek-Korjakowska and Tadeusiewicz [41], which consists of the following steps: image enhancement, lesion segmentation, border irregularity detection and classification. To find border irregularity, the authors translated the border into a function with peaks indicating the border irregularity. This is achieved by implementing a four step algorithm: (i) computing the bounding box of the segmented skin lesion; (ii) finding the boundary pixels lying on the lines that connect the center of the mass with the vertices; (iii) calculating the distance between the border and the edge of the image, which results in a function with an exact reflection of border irregularities. The signal is smoothed using a Gaussian filter in order to determine the ragged edges; (iv) finally calculating the derivative to find the local maximum points of the function, such that the local maximum is detected when the function crosses the zero point and the slope changes from + to −. The authors used a simple method to measure border irregularity, in which a simple semi-quantitative evaluation method is used to divide the lesion into eight similar parts such that the sharp abrupt cut-off in each part has a score of 1. Thus, a maximum score of 8 is obtained if the whole border is irregular, and a score 0 is obtained if the naevus is round with no ragged borders (authors in [48] divided the skin lesion into 8 segments and determined the pigment change in each segment for the measurement of border irregularity, with score ranging from 0-8). Zaqout [51] on the other hand found the compactness index: I = P 2 2πA for each segment and the border irregularity index was calculated as the sum of those indices). As a rule of thumb, melanomas tend to have scores 4-8 [42]. This is tested on 120 skin lesion cases with border irregularity less than 3 and 180 skin lesion cases with border irregularity greater than 4. The proposed approach achieves a 79% accuracy.
To determine border irregularity, Kasmi and Mokrani [46] divided the skin lesion into eight equal slices and approximated the sub-contour of each slice using a third-order spline function. A fitting error is then computed such that if the error is larger than (0.05× sub-contour length), the sub-contour is considered irregular. Each irregular sub-contour will be given a score of one, with the maximum score a lesion can have is eight.
After segmenting the skin lesion and extracting the lesion border using the Canny edge detector [59], Ali et al. [54] proposed a border irregularity measure that combines fractal dimension, Zernike moments and convexity, which are represented in a 27-value vector (Zernike moments produce 25 values). Fractal dimension is found using the extracted border, and Zernike moments and convexity are found using the segmented image. The extracted measures were then trained on a CNN (convolutional neural network) [115] and Gaussian naive Bayes ensemble, which is then used for the automatic detection (i.e. classification) of skin lesion border irregularity on new images). The approach achieves outstanding results, obtaining an accuracy, sensitivity, specificity and F-score of 93.6%, 100%, 92.5% and 96.1%, respectively. A similar approach was proposed in [55], where FuzzEdge [60] is used to detect the border, fractal dimension and convexity are used to represent the skin lesion border irregularity measure, and a fuzzy neural network is proposed to predict the lesion border irregularity based on the trained model on the extracted irregularity measure. This approach outperformed most of the state-of-the-art classification methods in general and its standard neural network counterpart in particular. However, it was more time-consuming when training the network.

IV. COLOR VARIEGATION
Color variegation refers to the presence of two or more shades of pigment (two or more colors) within the skin lesion border. Melanoma lesions often contain more than two colors as opposed to benign lesions which tend to be generally uniform in color. In particular, melanoma contains one or more of these six suspicious shades of color: white, red, light brown, dark brown, blue-gray and black. Fig.3 depicts a skin lesion and its dominant colors [52].
In [43], Umbaugh et al. proposed a color segmentation algorithm for the identification of color variegation in skin tumors. The algorithm is composed of six steps: (i) color averaging to reduce spatial data and reduce noise; (ii) masking out features such as ulcer and crust; (iii) decomposing the color space into different colors; (iv) filtering the results to facilitate in segmenting the image into color objects; (v) labeling each color object and finding its area; (vi) higher level processing, which is used to define color variegation based on two rules: (1) if the ratio of the tumor area while excluding ulcer, crust and shiny areas (defined in step ii of the algorithm) to the entire tumor area is less than 0.5, no color variegation is present; (2) for any object of size greater than 2mm 2 , if it is composed of two or more colors then color variegation is present. The approach is tested on 160 images and achieved a 73% accuracy in determining the presence/absence of color variegation.
Mimicking the way dermatologists determine color variegation, in which they locate the areas with homogeneous color and decide the number of colors based on the number and size of those zones, Tenenhaus et al. [44] segmented the skin lesion and proposed two unsupervised learning methods for describing color homogeneity, namely Kohonen map and K-means clustering. In Kohonen map, a random selection of 5 pixels from each tumor in the database is obtained to form a 5 × 5 Kohonen map in the RGB color space (25 neurons in the map), which in turn represents the variegation of colors in the lesion. In a single color lesion most of the pixels will be projected in the same region, while pixels of a multiple color skin lesion will be projected across different regions of the map. Color variegation is determined based on a 25-dimensional vector which is obtained by finding the proportion of pixels projected onto each of the 25 neurons of the map. The authors also used K-means to classify the pixels of the skin lesion where the number of clusters is set to the number of colors detected by the dermatologists, which is set to k = 4 as the authors found that the dermatologists didn't spot more than 4 colors in the lesion in most cases. A 16-dimensional feature vector is formed in this step (4 clusters × RGB color of each cluster centroid). The detection of the blue coloration is finally made using the Hue component in the HSV (Hue, Saturation, Value/brightness) color space. The Hue values fall in the [0, 360] range, and the blue color is coded by the proportion of pixels in the blue interval [200,250]. Color features are eventually represented by a 42-dimensional vector, which is fed to a KL-PLS based classifier as input for the detection of color variegation. The approach is tested on 227 tumor images resulting in 66% accuracy.
To find color variegation, Kalwa et al. [45] iterated through each pixel in the skin lesion HSV image and extracted the hue value of each pixel, eventually grouping pixels that have a hue value within a specified range together. However, a drawback of this approach is that the authors used trial and error to represent the HSV value equivalents of the melanoma suspicious colors.
She et al. [110] quantify color variegation using the normalized standard deviation of the red, green, and blue components of the lesion which are defined as: C r = σ r M r , C g = σ g M g , and C b = σ b M b , respectively. σ r , σ g , and σ b are the standard deviations of the red, green, and blue components of the skin lesion area, respectively, and M r , M g , and M b represent the maximum values of the red, green, and blue components of the lesion area, respectively.
Kasmi and Mokrani [46] measured the Euclidean distance between each pixel of the skin lesion and the six suspicious colors of melanoma (white, black, light brown, dark brown, red and blue gray). A pixel is said to belong to some color if the distance is less than a threshold T which is set as the half of the distance between the two extreme colors (white and black). A lesion is considered to contain a suspicious color if the pixels belonging to this color represent more than 5% of the skin lesion pixels. This approach was previously used in [47], [48] and [51]. The same rationale was used in [52] to determine the suspicious colors present in each skin lesion. However, the CIELab color space is used in [52] which is more representable of the human perception than the RGB color space. Moreover, the derived CIELab values of suspicious colors in [52] are based on the color distribution of the dataset used, making it more accurate in determining the melanoma suspicious colors. A range of colors is formed for black, dark brown and light brown, as opposed to only being represented by one value. On the other hand, the colors of white, red and blue gray are represented by one value as they are not present in the images used in the experiments. It is not apparent how the colors in [46], [47] are derived (apart from the white, black and red colors in [47] where the standard RGB values are used). The RGB value for the white color in [46] does not represent the actual color (another color is produced rather than the color of correspondence). Authors of those two studies used only one representative value for each suspicious color which might not be representative enough especially when we have different levels (shades) of the same color (i.e. light brown). As opposed to those two studies, Ali et al. [52] used Minkowski distance instead of Euclidean distance.

V. DIAMETER
Melanomas usually have a larger diameter than benign moles, which is around 6mm (i.e. the size of a pencil eraser). To find the diameter, authors in [48] measured the horizontal and vertical dimensions of the lesion as h = max(i) − min(i) and v = max(j) − min(j), respectively, and then calculated the diameter as D = max (h, v) in pixels, that is the diameter would be the maximum of the two dimensions. The resulting value is converted to millimeters based on the true size of the image. Kalwa et al. [45] measured the diameter using the equation D = 2αγ , where α represents the side length of the minimum area rectangle enclosing the skin lesion in pixels, and γ is the conversion factor from pixels to millimeters which is found using the imaging system parameters (i.e. focal length, distance from the object to the lens). Garnavi et al. [50] proposed greatest diameter and shortest diameter. The greatest diameter is defined as the length of the line that connects the two farthest boundary points while passing across the lesion centroid. The shortest diameter on the other hand is the length of the line that connects the two nearest boundary points and passes across the skin lesion centroid. In [51], Zaqout used the major axis length of the segmented lesion as a measure for diameter in pixels and then converted the result to millimeters. Ali et al. [52] used Feret's diameter [62], [63] to measure the skin lesion diameter, which is the distance between two parallel tangents at the contour skin lesion that are located on opposite sides of the object at an arbitrarily selected angle. To find the diameter, Messadi et al. [64] first determined the coordinates (x, y) of each pixel in the skin lesion, and then calculated the distance between each pixel pair. The maximum of those distances is considered the lesion diameter. However, the authors did not mention the conversion to millimeters which made their measure lack the representation of the actual diameter in the real world.
Diameter in [110] was calculated using the formula D = 2a, where a is the semi-major axis of the best-fit ellipse. The result (in pixels) is eventually converted to millimeters using the prior knowledge of image pixel parameters and the spatial relationship at a particular magnification.
It should be emphasized that segmentation is a crucial step before diameter measurement can take place. Moreover, a major limitation in the methods attempting to measure the diameter is finding the correct conversion factor to millimeters, which mainly depends on the original image size taken in the real world, a feature which is not always available especially when working with online image datasets that lack such information.

VI. FEATURE COMBINATION
Combining the ABCD features (e.g., AB, AC, ABC) has greater accuracy in determining suspicious lesions. Thomas et al. [102] have shown the sensitivity and specificity of each individual criterion in the diagnosis of melanoma (Table.1), in addition to the sensitivity and specificity of the combination of these criteria in the diagnosis of melanoma (Table.2).
She et al. [110] combined skin pattern features (skin line direction and intensity) with ABCD features to enhance classification performance. Before extracting the ABCD features, the authors used a snake-based algorithm [111] to detect the lesion area and form a binary image (i.e. segmentation); the lesion centre, orientation, and best-fit ellipse were determined from the binary image. The dataset used in the experiments was composed of 36 colored 24-bit images of size 230 × 350 (melanoma: 16, naevi: 20) which were converted to grayscale for skin pattern analysis. Classification was eventually carried out using individual features and a combination of features. In particular, the means of the skin line direction for the skin and lesion areas and their differences were calculated and a scatter plot of the line direction difference has been formed; the scatter plot would represent the separability used for classification. The area under the ROC curve evaluated to 0.84. Using the same approach but with the skin line intensity as the feature evaluated to 0.80. Using asymmetry, border irregularity, color variegation (red, green and blue components), and diameter as individual features evaluated to 0.66, 0.62, 0.54 (red), 0.76 (green), 0.78 (blue), and 0.62, respectively. The authors then combined the 8 features (skin pattern features and ABCD features) and used Principal Component Analysis (PCA) [109] to reduce the features used in classification to 2 features, resulting in an area under the ROC curve of 0.94.
Jaworek-Korjakowska et al. [103] proposed a software system for the detection of melanoma based on the ABCD rule. The system starts by converting the input colored image to a monochrome image using Otsu's method [104] and then utilizing DullRazor for hair removal. Image enhancement proceeds by blurring the edges and removing salt and pepper noise using the median filter, and tackling holes resulting from Otsu's method using different morphological operations (i.e. erosion, dilation); skin lesion borders are eventually found which represent the region of interest (i.e. lesion). The author then extracts the ABCD rule traits via different standard methods. Asymmetry is evaluated using the asymmetry index, irregularity is determined using the Harris corner detector [105], color variegation is analyzed using color segmentation based on multidimensional thresholding, and if the diameter is larger than 6mm the factor D = 5. After the ABCD features were extracted, the author used the Total Dermoscopy Score (TDS) to indicate the malignancy (i.e. melanoma) of the lesion, which is defined as: where the score ranges of A, B, C, and D are (0 − 2), (0 − 8), (1 − 6), and (0 − 5), respectively. A skin lesion is classified as melanoma if TDS > 5.45, a benign lesion if TDS < 4.75, and as a suspicious lesion if the TDS lies between 4.75 and 5.45. The system was tested on 50 lesions (benign: 20, malignant: 30) and achieved 87% sensitivity and 80% specificity. TDS was also used in [48] and [51], with the latter applied on the PH 2 dataset 1 and resulting in an accuracy, sensitivity, and specificity of 90%, 85%, and 92.2%, respectively. Kasmi and Mokrani [46] used TDS, but the D feature referred to (Different Structures) as opposed to Diameter, where pigment network [112] and geometrical properties of the lesion (fractal dimension, asymmetry index, circularity, elasticity) were used to evaluate the D feature. The approach was applied on 200 dermoscopic images (bemign: 120, melanoma: 80) with size 712 × 454 obtained from the EDRA Interactive Atlas of Dermoscopy [42], and resulted in an accuracy, sensitivity, and specificity of 94%, 91.25%, and 95.83%, respectively.
After pre-processing the skin lesion image and determining the lesion area, Ramezani et al. [107] extracted a group of features that represent the ABCD rule traits. Asymmetry was represented by a group of 32 features (i.e. orientation angle, asymmetry index), border irregularity by a group of 34 features (i.e. irregularity index, compactness index), color variegation by 72 features (i.e. mean, standard deviation), and diameter by 7 features (i.e. best-fit ellipse diameter, major diameter). The authors further add 42 features of lesion texture (i.e. contrast, entropy) extracted using Grey Level Co-occurrence Matrices (GLCM) [108]. To reduce the number of extracted features, PCA was utilized. This reduced the number of selected features from 187 to 13. An SVM was eventually used for classifying skin lesions into malignant or benign. The dataset used was composed of 282 macroscopic images collected from different online dermatology atlases such as Dermnet, Dermis and Dermquest atlases. The dataset included RGB images of 149 benign lesions and 133 malignant lesions ranging in dimension from 259 × 382 to 1186 × 1369 pixels. 70% of the data was used for training SVM and 30% for testing the classifier. That is, 197 and 85 images were used for training and testing the classifier, respectively. The approach resulted in accuracy, sensitivity, and specificity of 82.2±3.57%, 77.02±5.97%, 86.93±5.46%, respectively.
In [64], Messadi et al. pre-processed the skin lesion by removing the hair using the approach proposed in [49], then segmented the grayscale image using histogram thresholding and level sets [113]. The ABCD features were then extracted and fed to a multilayer perceptron [55] (the approaches authors used to extract the ABD features are discussed in the previous relevant sections. For analyzing the C feature, four parameters were selected from the 14 proposed 1 https://www.fc.up.pt/addi/ph2%20database.html by Haralick et al. [114], namely: correlation, homogeneity, energy, and contrast). 320 color images containing both benign and melanoma samples from DermNetNZ were used in the experiments. The multilayer perceptron was run for 100 iterations and resulted in accuracy, sensitivity, and specificity of 87.32%, 90.34%, and 33.29%, respectively.

VII. DISCUSSION
Several attempts have been made to automate the ABCD rule features to come up with more objective evaluations of suspicious lesions (i.e. melanoma). In measuring the A (Asymmetry) feature, there has been reliance on statistical measures such as convexity and the symmetry distance, which in turn depend on the segmentation performance and the real shape of the lesion to be measured, respectively. New measures for asymmetry evaluation have also been proposed, such as the asymmetry index, where a threshold is derived using this measure and lesions are subsequently classified based on this threshold. Some studies evaluate asymmetry geometrically by dividing the images into sectors and building their assumptions based on those sectors. Others specify two symmetry axes around which the tumor would be rotated, and then an estimation of shape and texture asymmetry is made based on the overlapping pixels. The use of descriptors to evaluate the shape has been involved, such as Fourier descriptors and SIFT. As proposed in [52], different measures have been combined together in a vector as a unified measure for evaluating asymmetry. Classifying lesions in either symmetric or asymmetric has been mainly carried out using threshold or machine learning approaches such as SVM and decision trees where such classifiers are being trained on the extracted measures, and the learned model is used to classify new/test lesions.
As a prerequisite to extracting asymmetry measures, some studies segment the skin lesion to form a region of interest (ROI) from which the measures are extracted and asymmetry is evaluated. The choice of the segmentation approach is thus crucial in determining the accuracy of the asymmetry evaluation. For instance, the multi-stage segmentation approach proposed in [33] provides poor results that degrade the performance of the evaluation as hair was present in the segmentation output. Small image datasets have been used in different studies until the emergence of public datasets such as ''ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection grand challenge datasets'' [55], [56] that enable the use of more images when evaluating different approaches geared towards the early detection of melanoma. Authors in [52] utilized images from this dataset in their work.
In determining the B feature (Border irregularity), border detection is considered a crucial prerequisite for characterizing this feature. Different methods have been used for border detection such as the radial search algorithm [31], Canny edge detector [54], [59], and FuzzEdge [55], [60]. In measuring border irregularity, most studies rely on statistical (e.g. area, perimeter) and geometrical (e.g. fractal geometry) features which have been extracted from the detected border. Ali et al. [54], [55] combined different irregularity features in a vector to form their border irregularity measure. In classifying skin lesions as possessing irregular borders, some studies use a threshold-like measure (i.e. irregularity index [36]- [38]), and other studies use a machine learning approach where the model is learned to predict border irregularity on new skin lesion images. Jaworek-Korjakowska and Tadeusiewicz [41] used a neural network as a classifier, and Ali et al. used a CNN and Gaussian naive Bayes ensemble and a fuzzy neural network in [54] and [55], respectively.
Different approaches have been proposed in determining the C feature (Color variegation), such as segmenting the image into color objects and using heuristic rules in deciding the presence/absence of color variegation. Unsupervised learning algorithms (i.e. Kohonen map, K-means clustering) are used to form color feature vectors, a KL-PLS based classifier is then used to detect color variegation based on those vectors. A trial-and-error approach is followed to determine color variegation. Other approaches determined the presence of a melanoma suspicious color by measuring the distance (e.g. Euclidean distance, Minkowski distance) between the pixel in the skin lesion and the six suspicious colors, such that the pixel was considered to belong to the color based on a threshold, and the lesion is said to contain the color if the pixels belonging to the color form a portion (i.e. more than 5%) of the skin lesion pixels. Ali et al. [52] used a range of colors to represent black, dark brown and light brown, as opposed to only being represented by one value. Segmenting the skin lesion before analyzing color variegation is beneficial as it narrows the search to a specific ROI, and eliminates the presence of any color outside the skin lesion that might otherwise be counted as a melanoma suspicious color. No machine learning based approaches have been used in the reviewed papers for determining color variegation.
In measuring the D feature (Diameter), the reviewed studies mainly rely on geometrical approaches in measuring the diameter in pixels, and eventually converting the result into millimeters based on a conversion factor. Such approaches suffer from a major limitation when it comes to representing the result in a way that would reflect the real world. Segmentation is also considered a crucial step before measuring diameter. No classifiers (i.e. threshold or machine learning based) have been used in the reviewed studies.
The standard pipeline of automatic dermoscopic image analysis consists of three main stages: image segmentation, feature extraction, and lesion classification. Fig.4 depicts the ABCD traits and the stages involved in each trait based on the reviewed papers.
Combining the ABCD features as opposed to using them individually improves melanoma detection accuracy, sensitivity, and specificity. This also applies to combining the ABCD features with other features extracted from the skin lesion.
As opposed to the hand-crafted features (i.e. statistical and geometrical features) which are apparently the type of features utilized while tackling the ABCD rule automation,  data-driven features derived from deep learning methods have been recently proposed in literature. However, the main drawback of using such features is their inability to detect fine structures (i.e. ABCD rule traits) [54]. The earlier attempts (to the best of our knowledge) in applying deep learning to melanoma detection were proposed in 2015 in [99], [100] (in Japanese) and [101]. Fig.5 depicts how published papers on using deep learning for melanoma detection increased in the period 2015-2017. Since 2018 the number has however been harder to track due to the dramatic number of papers published on the topic. The main papers published in 2016 can be referred to in: [65]- [79], and those published in 2017 in: [80]- [98]. Table.3 highlights 13 papers we chose from those published in the period 2015-2017 along with their year of publication, implementation frameworks used, and the details of the datasets utilized (i.e. size, image type). Papers chosen were those that had sensitivity and specificity values demonstrated and the main deep learning approach used in melanoma detection explained. Five studies used a dataset size less than 1000 images: [67]- [71], two studies used more than 1000 images: [65] and [66], five studies used more than 2000 images: [72], [80], [81], [88], [99], and one study used more than 100,000 images: [82].
The lowest and highest sensitivity values reported in the chosen studies are 0.51 and 0.95, respectively, with a pooled sensitivity of 0.76. The lowest and highest specificity values reported are 0.73 and 0.99, respectively, with a pooled specificity of 0.86. The pooled DOR (diagnostic odds ratio)  evaluates to 12.95; a small increase in the likelihood of the disease (+ = 3.82) and a small decrease in the likelihood of the disease (− = 0.41) have also been noticed. The higher the DOR the better the test. A test provides no diagnostic evidence if DOR = 1, while a test with DOR > 25 provides strong diagnostic evidence and a test with DOR > 100 provides a convincing diagnostic evidence. Using deep learning for melanoma detection thus shows poor diagnostic performance as evidenced by the DOR (12.95), which depicts that the odds of a positive test result is 12.95 times greater for someone with melanoma than without melanoma. This finding is confirmed by the likelihood ratio, where + = 3.82 means that the positive malignancy (i.e. melanoma) is 3.82 times more common in patients with melanoma than in those without melanoma. In other words, the patient's positive test result would be 3.82 times more likely to be seen in someone with melanoma than in someone without melanoma. On the other hand, − = 0.41 shows that a negative malignancy (i.e. benign) is 0.41 times more likely to be seen in patients with melanoma than in without melanoma. This poor diagnostic performance could be due to different factors like the small datasets and the quality of images used. The general accuracy of the tests is considered good referring to their AUC = 0.88 value (area under the receiver operating characteristic -ROC). The accuracy of the tests improve when the summary receiver operating characteristic (SROC) curve (Fig.6) moves to the top-left corner. That is, towards the point (1, 0) of the graph. Results also show good accuracy in terms of pooled sensitivity (0.76) and pooled specificity (0.86). The reason for the extra test results (red circles) shown in the figure (i.e. > 13) is due to the fact that some studies ( [81] and [99]) included more than one experimental result in their work.
Multiple skin datasets have been publicly available that could aid in diagnosing skin disease in general and in melanoma detection in particular. Table.4 provides a summary of such datasets along with their size and address. The main advantage of extracting the ABCD features using the automated methods discussed in this paper is the ability to provide an objective second opinion to the investigator (i.e. physician) which would otherwise be prone to subjectivity, especially that the ABCD features demonstrate fine structures of the skin lesion. The drawback however is that many approaches rely on the segmentation performance which could be degraded due to the presence of different artefacts (i.e. hair), affecting thereby the feature extraction process. Moreover, machine learning based approaches suffer from limitations in data availability especially when dealing with individual features (i.e. A and B). The approaches also lack the ability to reflect real-world measures such as in the diameter measurement.
Different challenges need to be addressed when developing melanoma detection approaches in general and automating the ABCD rule in particular. Although public datasets are available, there is still shortage in the availability of less quality skin lesion images that are mainly taken via mobile phone cameras as a result of the increase in melanoma apps usage; this type of images will introduce a variety of problems image processing and machine learning researchers need to handle. Datasets pertaining the different features in the ABCD rule need to be built especially with the much labor work accompanied with building such datasets (i.e. feature annotation). From a logistical perspective, there need to be more coordination between researchers and dermatologists (i.e. feedback) and the availability of more machine power (i.e. GPUs) especially for machine learning based solutions, as this would increase the pace of research outcomes significantly.

VIII. CONCLUSION
The subjectivity in interpreting the ABCD rule lead to different efforts in coming up with solutions that would provide an objective evaluation of the ABCD rule traits. The proposed solutions involved the automatic dermoscopic image analysis standard pipeline main stages, namely segmentation, feature extraction and classification. Some studies used machine learning based approaches in the classification stage, and we expect an increase in using such methods in future studies due to the traction machine learning has gained in recent years, in addition to its ability in solving many problems which would otherwise be infeasible using traditional image processing approaches. As opposed to using an individual ABCD rule feature, a combination of features could lead to better accuracy, sensitivity, and specificity in melanoma detection. More public datasets need to be available to enhance research outcomes in the domain, and such datasets need to also include the real world diameter conversion factor, especially when the proposed solutions measure the diameter in pixels and convert the result into millimeters based on a conversion factor which is not always available. More research should be done where mobile image photos are involved especially with the rise of melanoma apps, as opposed to the use of dermoscopic images; this also requires the development of light-weight fast algorithms. Pre-processing approaches that deal with artifacts (i.e. hair) need to be proposed when segmenting skin lesions and detecting their borders. We believe that this survey will serve as a starting point for researchers interested in automating the ABCD rule traits.