KEBOT: An Artificial Intelligence Based Comprehensive Analysis System for FUE Based Hair Transplantation

Robots and artificial intelligence technologies have become very important in the health applications as in many other fields. The proposed system in this work aims to provide detailed analysis of pre-op and post-op stage of FUE hair transplant procedures to enable surgeon to plan and assess success of the operations. In order to achieve this target, a robotic and vision-based system imaging and AI based analysis approach is developed. The proposed system performs analyses in three main stages: initialization, scanning, and analysis. At the initialization stage, 3D model of the patient’s head generated at first by locating a depth camera in various positions around the patient by the help of a collaborative robot. At the second stage, where high resolution image capturing is performed in a loop with the usage of the 3D model, raw images are processed by a deep learning based object detection algorithm where follicles in pre-op and extracted follicle positions (i.e. holes) and placed grafts in post-op is detected. At the last stage, thickness of each hair is computed at the detected hair follicle positions using another deep learning-based segmentation approach. These data are combined to obtain objective evaluation criteria to generate patient report. Experimental results show that the developed system can be used successfully in hair transplantation operations.


I. INTRODUCTION
Androgenetic alopecia (AGA) is the most common type of hair loss in males. Almost 75 % of males is affected by AGA during their lives and seek for treatment [1]. Minoxidil and finasteride are widely used therapeutics for the treatment of hair loss. However, hair transplantation surgery still remains as the gold standard solution for the management of AGA. Follicular unit extraction (FUE) is an extensively used hair transplantation surgery technique. In this technique Follicular units (FU) are harvested from the donor area directly through small circular incisions [2]. A successful FUE surgery depends on many factors including proper consultation, an adequate pre-surgery plan, quality of grafts and skin. Donor area management and recipient area design are The associate editor coordinating the review of this manuscript and approving it for publication was Aysegul Ucar . the main concerns in planning FUE surgery. Knowing the mean hair diameter, hair density in each donor area, size of recipient area and number of grafts required for an acceptable coverage are essential parameters for achieving the goals of pre-surgery plan [3]. If these numbers are not taken into consideration, risk of over harvesting donor area increases, and furthermore unsatisfying coverage of the recipient area might be the result. Coverage Value (CV) introduced by Dr. Erdoğan is the mathematical formula of the minimum acceptable coverage of the recipient site in 1cm 2 . This index is calculated by multiplying hair diameter with number of hairs located in 1 cm 2 [4]. Another determinant of hair transplantation planning is the identification of donor capacity. Donor capacity is the maximum number of follicular units that can be extracted without creating a density problem. CV also plays a crucial role in calculating the maximum donor capacity. VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Post-operative evaluation of a hair transplanted patients is also very important. Almost all the patients who have FUE surgery asks physicians how many grafts are placed, and the size of punch used in surgery. Up to date hair transplant surgeons do not have any proper instrument to answer these questions.
Robots in vision have many different applications areas [5], [6]. They were first used medically in 1985, can now be used in many different types of operations such as laparoscopy, neurosurgery, orthopedic surgery. In [7]- [9], studies are carried out about the accuracy of the robot which named Pathfinder used in neurosurgery operations. A robot system named Renaissance is proposed in [10] for brain operations such as spinal surgery. Several medical robots [11]- [13] are proposed to use in orthopedic operations. In [14]- [16] studies are carried out on the use of robots in laparoscopy operations.
In recent years, the use of robots has started in the hair transplantation. The robotic system called Artas, developed by Restoration Robotics, can perform hair transplantation [17]. This system includes a 7-axis KUKA robotic arm and multi-camera based stereoscopic vision system. Studies are carried out on the performance and accuracy of this system in [18]- [20]. The system used in Artas aims to perform FUE based hair transplantation in a semi-automatic way without in-depth analysis of whole head.
In recent years, artificial intelligence has achieved outstanding success in healthcare. There are lots of successful application in many sub-fields in healthcare as disease diagnosis in radiology field, drug development in medicine, robotic operation in surgery etc. The increasing of digital healthcare data and rapid development of new methods increase the use of AI in healthcare [21].
There are some AI technologies applied to healthcare. Machine learning is the most basic form of AI. Machine learning is using algorithms to learn from data and make decision or prediction about a specific task. In learning process, main aim is the fitting a model with defined features of data. Therefore, the performance of the model is directly proportional to the specified features. Deep learning is a sub-field of machine learning that mimics human perception inspired by human neural system as brain and the connection between neurons. Most deep learning methods use neural network architecture. These neural network models have many levels of features or variables that are used to predict outcomes. Main difference between machine learning and deep learning is the learning features stage. While the features to be learned for machine learning should be given explicitly, deep learning methods determine the most suitable features for the given data during the training phase [22].
These data can be 1-dimensional as sounds, 2-dimensional as images and 3-dimensional like videos. The applications and approaches to be used may differ depending on the data type. In this work, deep learning-based image processing approaches have been used since the images taken from the camera will be processed. In general, deep learning-based image processing approaches could be divided to recognition, object detection and segmentation. Recognition is the identifying the object present in images with its class label. Object detection approaches produces bounding boxes with its class label to localize objects in images. As a further step to object detection, segmentation approaches aim to generate pixel-wise masks for each object in images. Unlike object detection, it is possible to obtain shape information of objects with this technique. The state-ofthe-art object detection methods are based on deep convolutional neural networks. RetinaNet [23], M2Det [24], YOLO v4 [25] and EfficientDet [26] are the most popular and successful object detectors in this field. For segmentation approaches, SegNet [27], UNet [28], ERFNet [29] stand out with their success and processing speed in this area.
In this article, an automated vision based FUE based hair transplantation system is proposed. The main contributions of this work are as follows: • The proposed system is the first study that provides vision based comprehensive analysis ability for FUE hair transplantation.
• The proposed system is the first system that can cover whole head with its robotic arm and can capture every detail in the head with its camera setup.
• The proposed vision-based system is the first system that aims to detect all follicles in pre-op and all extracted follicles and all placed grafts in post-op stages of FUE based hair transplantation.
• The proposed vision-based system is the first system determines hair thickness in each follicle and CV for whole head.
• The proposed system uses deep learning-based techniques in this areas first time and achieve a good accuracy with thousands of manually labeled data generated in this work. The rest parts are structured as follows: In section 2, The initialization, scanning and analysis stages of the proposed method are explained in detail, respectively. Then, the results of the proposed system have been evaluated in Section 3, Lastly, the conclusion is summarized in Section 4.

II. PROPOSED SYSTEM
The proposed system in this work aims to provide detailed analysis of pre-op and post-op stage of FUE hair transplant procedures to enable surgeon to plan and assess success of the operations. In order to achieve this target, a robotic and vision-based system imaging and AI based analysis approach is developed.
Hardware components of the proposed system is depicted in Fig 1. Active IR Based Depth Camera is used to generate 3D model of the patient's head which is utilized to plan the route of the collaborative robot. The generated 3D model of the head is used to decide where high resolution image capturing will be performed. This data is also used to compute area of drawn region on the head. The Collaborative Robot is controlled by the Computation Unit using  the Robot Control Unit. Optical System and Light Source are employed to take pictures during scanning process. Operator Display and Patient Display allow operator and patient to monitor process. All system components are controlled by the Computation Unit. In KEBOT, the patient is positioned in specially designed chair where the head position is fixed. The imaging system consisting of two cameras and a light source aims to capture patient's head. In order to cover whole head at a very high resolution both cameras are attached to a collaborative robot by using a custom designed tool. A photograph of the developed system and close-up view of the tool is shown in Fig. 2  KEBOT system has two main usage: pre-op and post-op analysis. At the pre-op analysis stage, main purpose is to find out total donor capacity of the patient for an optimum operation plan. In order to obtain this information, it is required to count all follicles, number of hairs in each follicle, thickness of hairs and area of donor and recipient regions. Once all the information is obtained, it becomes possible to make an optimum operation plan. At the post-op analysis stage, it is required to obtain how many follicles are extracted from the donor region and how many of them are planted. If these data are obtained, it is possible to evaluate the performance of surgeon since the difference between this numbers reveals the number of unused or wasted follicles (i.e. total transections). KEBOT is designed to obtain these important pre-op and post-op parameters in accurate and automatic fashion.
At the pre-op stage, the developed system is to detect the hair follicles and calculate the thickness of these hairs in shaved head. Although it is possible to capture the whole head area in one shot, it is not likely to determine the hair follicles and to calculate thickness of hairs when 40-120µm shaft diameter is considered by making use of modern image sensors and optics. For this reason, the image of the head should be taken small region by small region to achieve required optical resolution. Thus, in this work, a 6-axis collaborative robot arm is used to ensure the movement of optical systems which consists of a depth and RGB camera and lighting. The depth camera is utilized to create 3D patient data to plan the route of collaborative robot and to perform area computations on the head. The RGB camera to take the 2D images of patient's head in a very high resolution so that in pre-op stage hair follicle can be detected and in post-op stage extracted and planted follicles can be determined. It is important to note that a special bi-telecentric lens is used in order to reduce the perspective effect to ensure accurate determination hair thickness.
The proposed system performs abovementioned analyses in three main stages: initialization, scanning, and analysis. The flowchart of processing stages of the proposed system is given in Fig. 4. First, at the Initialization Stage the 3D model generation is carried out at first by locating depth camera in various positions by the help of the collaborative robot. At the second, the Scanning Stage, once the 3D model is generated an operator determines ROI in 3D data and this cropped 3D data is used to generate the route for high resolution image capturing. The generated 3D data is also utilized to compute area of drawn regions in 3D space.   with object detection results are handled analysis software where hair thickness is computed at the detected hair follicle positions using another deep learning-based segmentation approach. After the hair thickness is obtained, all data are integrated to obtain objective evaluation metrics to generate patient report.

A. KEBOT INITIALIZATION AND SCANNING PROCESSES
Collaborative robots generally operate on routes which are manually taught. However, since shape and size of the patient's head differs, the route that the robot will follow during the 2D image acquisition must be computed specifically for each patient since the distance between the RGB camera and the captured head region must be within a certain range. This is mainly because of limited depth of field of the imaging system. Thus, we generate 3D model of the patient's heads by making used of the depth camera at the initialization stage and the route for the optical system for scanning procedure is computed according to the 3D model of each patient. The block diagram of the scanning process which includes Initialization and Scanning Procedure stages is given in Fig. 5.
The purpose of the initialization stage is to generate 3D model of the patient's head and to determine the areas to be captured on the head with the help of this 3D data. In the Scanning Procedure, the inverse kinematic calculations of the collaborative robot are performed for the positions determined in the previous stage, and image capture and saving operations are carried out. The data collected at this stage are sent to the AI Based Analysis System.
In the first stage of the initialization stage, a 3D model of the patient's head is created by moving the depth camera on a pre-determined fixed route. An example 3D model created after this operation is given in Fig. 6.
While creating a 3D model by depth cameras, these cameras employ the starting point as the center of the coordinate space. Therefore, the 3D model created, and the origin of the robot are different from each other. In the next step, the coordinate system of the 3D model is transformed considering the position and orientation information of the robot's first position on the fixed route. In the next step, region of interest (ROI) is determined on the 3D model of the patient by manually marking certain points with the help of an operator. With the camera and lens system used, 2.5 cm horizontal FoV (Field of View) and 2 cm vertical FoV are obtained. Considering these distances, the points where the camera will take capture on the 3D model are determined by sub-sampling (Fig. 7).
In Fig. 7, the red dots represent the center points of the areas where the camera will take images. Optical systems that uses bi-telecentric lenses must have a fixed distance to the area they will capture due to their internal structures in order to avoid blurry (out of focus) images. The optical system used in this work must be 10 cm away from the object to obtain in-focus images. Our optical system needs to located perpendicular to area to be captured to eliminate possible blurry images and perspective effects.
Thus, orientation of the area (2.5cm × 2cm) of the head that will be captured has be known. In order to calculate the orientation, the triangles that make up the 3D model are used. Using this distance, orientation and the points calculated in the previous step, the positions of the robot to capture images are computed (see Fig. 8). Using these positions and orientation values, robot joint angles are computed with inverse kinematic and sent to the robot controller. After the robot updates its position, the light source is turned on, the image is captured and stored.

B. AREA MEASUREMENT
One of the information that need to be used for the CV calculation is the area measurements of the regions where hair follicles extraction and plantation will be performed. Areas such as occipital, parietal, frontal etc. are marked by the doctor using a whiteboard marker before the operation. An example is given in Fig. 9.
In order to calculate the CV, the regions marked by the doctor are marked manually on the colored 3D model with a software module developed in this work. In the next step,  the area of these marked surfaces calculated by the sum of the areas of the triangles that make up the three-dimensional surfaces. Example screenshots of the GUI showing this process are given in Fig. 10.

C. AI BASED ANALYSIS SYSTEM
The proposed deep learning-based hair analysis system consist of three main stages: deep learning-based object detection, deep learning-based hair thickness calculation and metrical analysis. Results of first two stages are combined at the last stage. As explained before, KEBOT system can perform pre-op and post-op analysis using AI based algorithms.

1) DEEP LEARNING BASED OBJECT DETECTION APPROACH
In the developed KEBOT system, detection and counting hair follicles and post-operative placed graft and scar traces are considered as an object detection problem. In order to solve this object detection problem, a deep learning-based approach is used.
The KEBOT system collects images in 18 Mpixel resolution from patients during scanning for training and test. Thousands of pre-op and post-op images are collected from real patients and these are manually labeled by the nurses who work and expert in hair transplantation. The labeled examples are shown in Fig. 11. There are approximately 250 to 450 follicle or extraction trace samples to be detected in every single image. There are 7 classes in pre-op images and 2 classes in post-op images. In pre-op these are the numbers between 1 to 7 and corresponds to the number of hairs in that hair follicle and in post-op these 2 classes correspond to placed graft and scar traces.
RetinaNet [23] M2Det [24], YOLO V4 [25] and Efficient-Det [26] are recently proposed state of the art deep learningbased object detectors. Their performance is reported in many     different applications. These methods are able to provide a good balance between processing speed and accuracy. Among these methods RetinaNet is preferred for the proposed system.
RetinaNet, a deep learning-based object detection network with ResNet-101 backbone is used to detect follicles and transplanted graft and scar traces. RetinaNet network architecture is shown in Figure 12. Two different networks have been trained for post-op and pre-op detections. Both networks used the pretrained weights of ImageNet for ResNet-101 backbone at start and uses Focal Loss as a loss function. Since the 18Mpixel resolution images are large for the input of the object detection networks, the input images are resized to 1000 × 1333. The output class numbers are set to 7 in pre-op training and set to 2 in post-op training. Networks hyperparameters and anchor sizes are updated due to small size the follicles and transplanted graft and scar traces. Training and test images are resized as they are large for network and virtually augmented using horizontal flip.

2) DEEP LEARNING BASED HAIR THICKNESS CALCULATION APPROACH
In the KEBOT system, hair thickness calculation approach has two steps as deep learning-based segmentation and novel thickness computation method. First, the deep segmentation approach is used to separate hairs from background to measure the hair thickness. Labeled pre-op object detection images are used to create a segmentation dataset. Every box that labeled in object detection dataset are cut from the whole image and label-wise annotated manually by the nurses. Pixel-wise segmented samples are shown in Figure 13. The segmentation dataset contains 2 classes. Segmented hair area pixels are labeled as hair and remaining pixels are labeled as background.
SegNet [27], UNet [28], ERFNet [29] are recently proposed state of the art deep learning-based segmentation methods. When evaluated in terms of performance and processing speed, SegNet stands out among these methods and is used in this work. SegNet a deep convolutional encoder-decoder network is used to segment the given hair samples. SegNet architecture is shown in Fig. 14. The hyperparameters of the network have been updated and the input image size of the network have been set to be 120 × 120 pixels due to small size of the input images. The network uses the ImageNet pretrained weights for VGG16 backbone. Training and test images are resized according to the input dimensions of the network.
After segmentation step, in order to compute the hair thickness in real-work dimensions (i.e. in micrometer scale), the pixel thickness of the hair strand should be computed from segmented images. A novel algorithm has been developed for hair thickness analysis for this purpose. Only single hair strands are used in computation to prevents possible errors originating from the occluded hairs in a follicle.
In the first step of this procedure, the segmented image is converted to a binary image. The pixels that cover the hair strand take the value ''1'', and the others take the value ''0''.
Then, distance transform [30] is applied to the binary image and the pixel distance values of each pixel in the image are obtained. In the image obtained after the distance transform, each line in the image is examined with a certain step interval. The maximum distance values in the relevant line are obtained. Using the median of the obtained values, very large and very small values that may be erroneous are eliminated and the thickness is obtained by calculating the mean of the remaining values. Since this value is in pixels, it is multiplied with a pixel to micron conversation value named with Camera Pixel Physical Resolution (CPPR). This conversation value is calculated using FoV and resolution information of camera and lens system. The sample workflow of the algorithm is shown in Fig. 15. Fig. 15(b) shows segmented binary image and Fig. 15(c) shows distance transformed image with examination lines. This process is repeated for each single hair follicle and the hair thickness of the patient are calculated. A thickness histogram is created for each area (Left-Right 200468 VOLUME 8, 2020  Temporal, Left-Right Parietal and Occipital) using the thickness values of each calculated hair follicle and the values that could be erroneous in the lower and upper parts of the histogram are eliminated.

3) INTEGRATION AND METRIC COMPUTATIONS
Detected hair follicles, calculated hair thickness, and measured area size are integrated and used in performance calculation at the last stage. For performance calculation FU/cm 2 , Diameter size (hair thickness) and Calculated Density is used in KEBOT system. FU/cm 2 is total follicular unit in a 1 cm 2 area. Diameter size is average hair thickness of the patient. Calculated Density is average hairs in one follicle. Using these 3 metrics, the CV, which is a general performance evaluation criterion, is computed using the equation given in (1).
CV provides information as quality of the donor area, required grafts count for minimal coverage in a certain area and total donor capacity for a patient. Since this information are important, CV should be calculated correctly.
First FU/cm 2 is calculated using area computation (Section 2.2) and total detected follicles in this area (Section 2.3.1). Area sizes are determined using the drawn area by the operator on 3D model (see Fig. 10) which is the areas drawn by the doctor using a board marker. These areas are left-right temporal, left-right parietal and occipital. Then, detected follicles in these areas are determined and divided to the corresponding area size. As a result of this operation FU/cm 2 is computed for each area separately.
Secondly, Calculated Density is computed from same follicles used in the first step. Total hair count in these follicles are divided to follicle count to obtain this metric. At last, Diameter value is obtained from hair thickness calculated on the hairs in the selected area. These hair thicknesses are calculated as described in Section 2.3.2. In this step, only the value belonging to the selected area is used. Finally, all these three computed parameters are used in CV computation for each area that is being analyzed. At the end of this processing pipeline, a report for each patient is generated. An example report is shown in Fig 16.

A. DATASET GENERATION
The images used for object detection and segmentation algorithms are created using real patient's data scanned by the KEBOT system. KEBOT system captures hundreds of highresolution images from each patient during pre-op and postop scanning. Each image of the patients is examined, and a few appropriate images are selected for the training of the object detection algorithm by considering diversity of samples. Selected images are marked manually by the nurses using a custom designed object detection labeling program.   GUI of the custom designed object detection labeling program is shown in Fig. 17. Nurses annotate the hair follicles for pre-op and the areas where transplantation and extraction are carried out for post-op using the labeling program. The labeling program saves the coordinates and class label of the marked objects in each image to a log file.
The hair follicles in the pre-op images labeled for the object detection dataset are cut from the images and saved. These saved images are examined, and appropriate samples are selected for the segmentation algorithm. All selected samples are manually segmented as hair and other non-segmented parts automatically segmented as background by the nurses using a custom designed labeling tool. Sample images from segmentation labeling process are shown in Fig. 18. The labeling tool saves the segmented data as a new image which has a class label in every segmented pixel.
The total number of images are used for object detection and segmentation in the KEBOT system is given in Table 1. Detailed number of hairs in hair follicles and extracted and planted places numbers, which means class labels, are given in Table 2 and Table 3 for Pre-op and Post-op dataset, respectively.

B. TRAINING
Object detection and segmentation algorithms that used for analysis in KEBOT system have been trained offline on a      be detected and segmented in one image, a powerful system is needed for training. The workstation used in training and test has 3.3GHz 10 Core intel Core i9-7900X Processor, Titan XP Graphics Card and 64 GB RAM. Since the number of images in the data sets increased over time, many trainings are performed in different number of training samples.
In the first few training processes, parameters of deep networks have been optimized to give targeted object detection and segmentation results. As the number of images and the variety of data increased, new trainings are carried out and it is aimed to increase the performance of the algorithms. The best pre-op and post-op training loss curves are given in Fig.19.
RetinaNet object detection network is trained 200 epochs both for pre-op and post-op and learning rate is initialized 1 × 10.3, which is the divided by 10 at 160 and again at 190 epochs. Both networks trained with batch size 2. SegNet   Table 4 and Table 5. Post-op results are calculated based on the number of placed and extracted detections and performance evaluation is given in Table 6. When Table 4 is examined in detail, it is seen that FUs can be detected almost without any problems. However, as can be seen from the maximum difference in CD value in this table, the classes of the detected FUs can be miss classified with a high rate of error in some patients. The main reason for this is that the patient generally moves excessively during scanning which results in blur effect in captured images. Table 5 shows that the proposed system can achieve successful results regardless of the head structure and region. In the results of Table 6, EG and PG are below 5% which means that the patient can be analyzed at a neglectable error level.
For the training process of the proposed segmentation approach, 1000 test images annotated manually by the nurses. After the segmentation performance reaches a certain level, real patient tests are started. CPPR parameter is 5.12 for the current KEBOT setup. In the process of determining the real hair thickness of the patients with segmentation algorithm,     all these 270 hair strands have been obtained using a scanning electron microscope (SEM). Example measurement of hair samples are shown in Fig. 20. Then, these three patients are scanned by the KEBOT system and the thickness values obtained as a result of the scanning are compared with the thickness values are found by SEM. Performance evaluation of segmentation approach is given in Table 7. In Table 7, SEM column considered as ground truth and difference values are computed using this value. As seen from table, maximum difference percentage is lower than 5% and, this is considered an acceptable difference that will not change the patient's assessment.

IV. CONCLUSION
In this article, a method and system for comprehensive analysis for FUE based hair transplantation are proposed. The proposed system consists of a 6-axis collaborative robot arm, one depth and one RGB camera, lighting source and a processing unit. The images captured by the RGB camera is processed by deep learning-based algorithms in order to detect follicles count, extracted count and placed counts, and determine hair thickness. Then all obtained information are used to compute CV to enable doctor to plan the operation. The post-op analysis provided by the proposed systems make the assessment of the operation possible. Extensive experiments and test reveal that the proposed system is able to provide valuable information regarding to FUE based hair transplantation operations.