Dental Caries Detection Using Score-Based Multi-Input Deep Convolutional Neural Network

Panoramic and periapical radiograph tools help dentists in diagnosing the most common dental diseases, such as dental caries. Generally, dental caries is manually diagnosed by dentists based on panoramic and periapical images. For several reasons, such as carelessness caused by heavy workload and inexperience, manual diagnosis may cause unnoticeable dental caries. Thus, computer-based intelligent vision systems supported by machine learning and image processing techniques are needed to prevent these negativities. This study proposed a novel approach for the automatic diagnosis of dental caries based on periapical images. The proposed procedure used a multi-input deep convolutional neural network ensemble (MI-DCNNE) model. Specifically, a score-based ensemble scheme was employed to increase the achievement of the proposed MI-DCNNE method. The inputs to the proposed approach were both raw periapical images and an enhanced form of it. The score fusion was carried out in the Softmax layer of the proposed multi-input CNN architecture. In the experimental works, a periapical image dataset (340 images) covering both caries and non-caries images were used for the performance evaluation of the proposed method. According to the results, it was seen that the proposed model is quite successful in the diagnosis of dental caries. The reported accuracy score is 99.13%. This result shows that the proposed MI-DCNNE model can effectively contribute to the classification of dental caries.


I. INTRODUCTION
Dental caries is one of the most common dental diseases. Dental caries result from a complex interaction between acid-producing bacteria that adhere to the dental and fermentable carbohydrates. Acids in dental plaque can demineralize enamel and dentine in the cracks and smooth surfaces of the dental. The earliest visual sign of dental caries is the white The associate editor coordinating the review of this manuscript and approving it for publication was Tyson Brooks. spot lesion. If demineralization continues, white dot surfaces get pitted, causing small holes called cavities to form. It is particularly typical in children, teenagers, and older adults. However, anyone who has teeth, including babies, is likely to have dental caries. If cavities are left untreated, they can grow and affect the deeper layers of your teeth. They can cause severe toothache, infection, and dental loss [1], [2]. Periapical radiographs (peri means ''around'' and apical means ''end of the dental root'') record images of the outlines, location, and mesiodistal size of the teeth and surrounding tissues.
The intraoral periapical examination obtains the view of the entire dental and surrounding structures [3]. Intraoral periapical radiography is a widely used intraoral imaging technique in dental radiology and provides considerable information about the teeth and surrounding bone. The film presents vital information to aid in the diagnosis of the most common dental diseases, especially dental caries, dental abscesses, periodontal bone loss, or gum disease, and shows the teeth and surrounding alveolar bone, all dental coatings, and dental roots. The application can supply an opportunity to identify additional significant findings, including the status of restorations, impacted teeth or broken dental fragments, and variations in dental and bone anatomy [3], [4].
Caries detection based on panoramic and periapical imaging techniques is performed manually by dentists. Mostly the caries dental diagnosis seen on the radiography cannot be made correctly because the existing dental caries can be overlooked due to the physician's inexperience or intense patient load. This situation can lead to the progression of caries, advanced dental infections, and dental loss. Automatic systems, which have been developed based on machine learning and image processing techniques to avoid these negativities and facilitate dentists' diagnoses, have gained considerable importance. Dental caries, periodontitis, interdental bone loss, gingivitis, impacted teeth, and cysts are some dental diseases. In the literature, there are few studies for detecting dental diseases. In these studies, the subjects such as caries detection (see 2nd literature review) and cysts detection [23]- [25] have been discussed. In most of these studies, traditional machine learning-based feature extraction methods were used to detect dental diseases [5], [7], [10], [11], [13], [15], [17]. In other studies [6], [8], [9], [12], [16], segmentation and classification processes were performed using deep learning models. Besides, there are few studies based on deep CNN architectures on dental x-ray images. Also, applications on dental caries detection using deep CNN architectures are even more limited in quantity and inadequate in this area.
This study presented a novel score-based multi-input deep convolutional neural network ensemble to detect dental caries using periapical teeth images. The proposed model used image processing techniques to sharpen raw caries images. Later, the multi-input deep convolutional neural network was designed using raw and processed images, and the training process was performed. Finally, the Softmax layers of the multi-input CNN architecture were combined with the score level-based algorithm. In experimental studies, 340 periapical images, including caries and non-caries, were used. These results have revealed that the model proposed for the classification of dental caries will provide remarkable contributions to dentists' performances.
This work presents a new dental caries detection model based on the multi-CNN ensemble. The major contributions of the proposed model are: • The sharpening filter and intensity colormap procedures, which are used in this study, further highlight the decayed area in the unprocessed dental images.
• Although there are studies on dental caries in the literature, there is no publicly available data set. In this paper, a new dataset consisting of 340 images (caries and noncaries) was presented, and this dataset was made freely accessible for researchers.
• Pre-trained AlexNet architecture based on the transfer learning approach in dental caries detection was adapted. In this study, instead of starting the training process with random weights using a data set containing a limited number of images, the learned weights of pre-trained architecture were used. In the experimental results, it was observed that the transfer learning based AlexNet model used for the proposed model increased the success by approximately 10% compared to the Training from scratch approach.
• The proposed score-based multi-input CNN model has achieved notable success in detecting dental caries with an accuracy score of 99.13%. This result showed the usability of the proposed model in real-time applications. The purpose of the present study was to determine the ability of deep convolutional neural networks to assist dentists in the automatic diagnosis of dental caries based on periapical images.

II. LITERATURE REVIEW
In the introduction part, general studies on dental disease detection were mentioned. Few studies on the detection of dental caries are available in the literature, and in these studies, commonly panoramic and periapical imaging systems have been used. In the research, pre-trained deep models have been used commonly, besides traditional feature extraction methods. Geetha et al. obtained dental X-ray images using the Laplacian filtering, window-based adaptive threshold, and back-propagation neural network. The resulting new views from a few statistical feature extractions were classified by the back-propagation neural network. A success of 97.1% was achieved in the caries detection using periapical 105 dental x-ray images [5]. Prajapati et al. used a transfer learning approach based on the pre-trained VGG16 model on 251 periapical dental X-rays. In the study performed for three distinct classes, 87.5% performance was obtained in the images of caries from the fully trained network architecture [6]. In their study, Singh and Sehgal proposed an automated caries detection system based on Radon Transformation (RT) and Discrete Cosine Transformation (DCT). The selected features were extracted using the PCA technique. Finally, the model was applied to the Random Forest classifier, and an accuracy of 86% was obtained [7]. Casalegno et al. have proposed a method for the detection of caries in dental images obtained with near-infrared transillumination (TI) imaging technology. With the proposed convolutional neural network architecture, ROC areas of 83.6% and 85.6% were obtained for occlusal and proximal caries lesions, respectively [8]. Cantu et al. conducted a study on the detection of caries lesions in 3686 bitewing radiographs. The U-Net convolutional neural network architecture was used on dental caries. The proposed neural network showed an accuracy of 80% [9]. In the study by Datta et al., special neutrosophic domain-based features were used. This proposed feature extraction method was obtained by combining certain features. Caries was detected from 120 periapical X-ray images using the active contour technique. As a result of the experimental studies, the performance was over 92% [10]. Tuan et al. proposed a hybrid approach consisting of segmentation, classification, and decision-making phases in their studies. A segmentation method based on semi-supervised fuzzy clustering is used. Besides, classification was carried out with a new graphic-based clustering technique. The method's performance recommended for five distinct carries classes has been observed as 90% [11]. Lakshimi and Chitra conducted a study to detect caries using 1700 panoramic dental x-rays. Performance results were compared with traditional segmentation techniques and deep CNN architecture. A success of 98.6% was achieved with the graph cut and deep CNN method [12]. The study by Naam et al. aimed to facilitate the identification of caries for experts in panoramic dental x-ray images and increase their accuracy. Multiple morphology gradient algorithms have been developed, besides wellknown approaches in image processing techniques such as cropping, morphology dilation, and morphology erosion. The improved pre-treatment stage facilitated the caries detection by clarifying the dental x-ray image better [13]. Oprea et al. proposed a method to examine the size of the existing caries lesion and then classify the type of caries found in dental radiography. Segmentation of the dental was performed using the edge detection method in the x-ray. Later, they performed a study by determining the pixel number of caries (black area) affected on the dental and classifying them according to their size [14]. Obuchowicz et al. made comparisons in caries detection by using specific texture feature mapping methods. Original images were pre-processed by applying the Histogram Stretching (HSTR) algorithm. As a result of the comparison in the texture feature map methods, it was observed that the most useful information was provided by using the energy parameter. Besides, it was stated that the entropy parameter also performed well in visualizing the areas affected by caries [15]. Lee et al. used GoogLeNet Inception v3 CNN architecture to detect dental caries from periapical dental X-ray images. In the proposed study, a periapical data set consisting of 3000 images were used for 2400 training and 600 tests. In the results, the success rate of premolar, molar, and both premolar and molar caries models were 89%, 88%, and 82%, respectively [16]. Naam et al. aimed to improve the quality of X-ray images in determining proximal caries by experts. In the dataset they created, there are dental x-ray images containing 27 proximal dental caries. With the proposed Multiple Morphological Gradients method, the object edges have been clarified by producing clear images. The results suggest the model can be used to define appropriately proximal caries and progression levels [17]. Leo and Reddy [18] proposed a hybrid model consisting of Artificial Neural Network and Deep Neural Network techniques. In experimental studies, they used 300 training and 180 testing dental x-ray images. The proposed system yielded a 96% success for dental caries classification [18]. Huang and Lee performed a study on caries detection using optical coherence tomography (OCT) images as a different imaging technique. A total of 748 single tooth images were extracted using the cropping process from 63 OCT tooth images. In experimental studies carried out based on different CNN architectures, 599 images were used for training and the rest for testing. As a result, they achieved a 95.21% success rate with the ResNet-152 architecture [19]. Bui et al. proposed a hybrid model for the detection of caries based on support vector machine (SVM) classifier using deep and geometric features. In experimental studies, they used 229 carries, and 304 non-caries images. As a result, the accuracy score of the proposed system was obtained as 91.7% [20]. Majanga and Viriri developed a system based on morphological methods and deep learning. In the preprocessing stage, the boundaries of tooth images were determined by using thresholding methods with Gaussian blur filters. Later, the classification of dental images (caries/non-caries) was carried out by using a certain threshold value with features obtained based on deep learning. They used 11,114 tooth images obtained using data augmentation methods based on periapical dental X-ray images. As a result, the proposed model for caries detection was 97% successful [21]. Vinayahalingam et al. carried out a study based on the evaluation of the classification performance of deep learning architectures using panoramic x-ray images. 400 cropped panoramic x-ray images were used for training CNN architectures. In experimental studies, it was observed that they achieved 87% performance with the MobileNet V2 architecture [22].

III. PROPOSED METHODOLOGY
This study presented a novel score-based multi-input CNN ensemble (MI-DCNNE) to detect dental caries. The proposed system consists of three phases: Pre-processing, Deep Convolutional Neural Network, and score-based fusion. In the preprocessing stage, filters are used to clarify the panoramic raw images. Later, a multi-input Convolutional Neural Network Model based on a pre-trained deep model is designed. In the last stage, the developed multi-input CNN model is combined at a score level. The general structure of the proposed system, including all these stages, is given in Figure 1. The stages composing the proposed system's general structure and dataset are detailed in subtitles.

A. DATASET
In this study, images obtained from two private oral and dental health clinics using periapical radiography devices (Planmeca Intra, Helsinki, Finland), (MyRay, Imola, Italy), as in retrospective studies, were evaluated. A total of 380 periapical images belonging to 310 patients were analyzed. 40 images including blurred images or images with fractures, cysts, and infection were excluded from the study. 340 periapical images out of 380 images were included in the study. A data set containing periapical radiography images obtained from patients was created. These radiographs included images of 150 carious teeth and 190 non-carious teeth. This dataset was obtained under control by a dentist. These images were gray channels and variable sizes. Figure 2 shows some samples of dental periapical images from the dataset.

B. PRE-PROCESSING
In the current study, panoramic images were used to classify dental caries. Several image processing techniques were applied to these raw images. This process has clarified the carious areas in the dental images and increased the classification performance. To this end, a sharpening filter and intensity colormap were applied for panoramic raw images, respectively. As a result, the same size new images were obtained. An example of this process is given in Figure 3.
As seen in Figure 3, the sharpening filter was applied for raw gray images as [−1 −1 −1; −1 9 −1; −1 −1 −1]. Then, the intensity values of the images were adjusted. In this way, the contrast of the new image was increased. Eventually, the caries zone was observed to become sharper in the recent pictures obtained as a result of filtering and density value adjustment processes.

C. DEEP CONVOLUTIONAL NEURAL NETWORK
A Convolutional Neural Network (CNN) has a multi-layer network structure trained in a specific system for the process of feature extraction and classification. It replaces time-consuming traditional methods of feature extraction and classification. The CNN architecture has a basic structure consisting of the input layer, multiple hidden layers, and output layers. Besides, the hidden layer consists of convolutional layers, pooling layers, and fully connected layers. High-level features obtained by filtering from input images are processed by convolutional and pooling layers. If we use a two-dimensional image as input, a two-dimensional kernel is used. Convolution is represented using the parameters shown in Equation (1).
where, the feature map of the previous layers is represented by the expression x l−1 i . M j , k l ij and b l j represent input map selection, convolution kernel and trainable additive bias terms, respectively. The Convolutional process is passed through an f activation function [26]. Filters are usually moved over the image with the specified stride (s). For an input image with width A, height B and channel C (A m × B m ×C m ); with r ×r size C 0 filters, the output volume will be VOLUME 10, 2022 A 0 × B 0 × C 0 . As a result of the filtering, the new dimensions shown in Equation (2) are obtained; where p represents the extra pixels to be added to the input matrix. The activation function enhances the nonlinearity in the network. Many studies in the literature use the Rectified Linear Unit (ReLU) function shown in Equation (3); Pooling, also known as the sub-sampling layer, helps regulate over-compliance. The dimensions of output maps are reduced by using maximum, minimum, and average values. Finally, the full connect layer reduces the multidimensional property map to a one-dimensional vector [27]. In this study, pre-learned weights of the AlexNet architecture for each of the multi-input images and a transfer learning approach to adapt this architecture in dental caries detection have been used. In this process, new fully connected, Softmax, and classification layers have replaced the last three layers of pre-trained AlexNet architecture. Additionally, the proposed model has used a score-based fusion process in the Softmax output layers of multiple CNN architectures. The general structure and characteristic features of the proposed model are detailed in Table 1.

D. SCORE-BASED FUSION
In the proposed system, an MxN output matrix is obtained from the softmax layers in deep CNN architectures applied for each multi-input image [28]. Here, N and M are represented as the number of classes and the number of test sets. These CNN models were trained individually with transfer learning fashion.
Later, Equation 4 is used to calculate the new score value; where A represents the score generated by the CNN-based models. A new score in MXN size was obtained using Equation 4. Later, using the new score value, the class label is determined according to the maximum value of each line (M). In the proposed system, two-CNN models based on two input images are used. In this context, the pseudocode of this approach is demonstrated in Algorithm Table 2.

IV. RESULTS
The experimental works were carried out on a computer equipped with the NVIDIA Quadro M4000 GPU with 8GB RAM. For all coding, MATLAB software was used. The dental periapical image dataset was collected in private dental clinics. All ethical issues were guaranteed. In this study, a novel score-based multi-input CNN ensemble (MI-DCNNE) model was proposed to detect dental caries. The weights of pre-trained AlexNet architecture were used for multiple entry images in the proposed model. Later, the softmax layer of the multi-input CNN model based on the score level-based algorithm was combined and the class tags were determined. The deep parameters used in the experimental studies for training the CNN architecture are given in Table 3. As seen in Table 3, in experimental studies, the epoch number and batch size were investigated in values between 5-20 and 8-16, respectively. In these empirical studies, 70% was used for training and the rest for testing. This random separation was done only once, and all experimental studies used the same training/testing datasets. Also, the test procedure was repeated 100 times for the results of all experimental studies. The AlexNet architecture based on raw images, the AlexNet architecture based on filter approach, and the proposed architecture performance have been calculated separately by using these processes. These results are given in Table 4. The minimum, maximum, mean, and standard deviation values of the results obtained from the AlexNet, Filter-AlexNet, and proposed model through 100 times of running are given in Table 4. According to these results, the maximum accuracy score for the AlexNet model based on raw images and filtered images was 98.26% for both methods. As seen from Table 4, the proposed score-based multiinput CNN ensemble model obtained a 99.13% accuracy score. Additionally, the other performance measures such as sensitivity, specificity, precision, and F1_score values were 98%, 100%, 100%, and 98.99%, respectively. The confusion matrix and ROC diagram of this accuracy result is also given in Figure 4.
As shown from the confusion matrix, the proposed model achieved a 100% accuracy score for the non-caries class and 98% in the dental caries detection. Besides, the training accuracy and loss curves of the proposed method was given in Figure 5. As seen in Figure 5, the training accuracy was around 75% in the beginning and it dropped to almost 40% accuracy score in the first epoch. And, it suddenly increased through 90% accuracy score in the first epoch. The training accuracy curve increased to almost 100% accuracy score after fourth epoch. Similarly, the loss curve was around 1 in the beginning of the training procedure and it increased to around 3 in the first epoch. It gradually decreased through zero after fourth epoch.
In addition, to achieve interpretability of proposed model, we investigated network predictions using the gradientweighted class activation mapping (Grad-CAM). The activation map for carious images is given in Figure 6. As seen in Figure 6, four test images and their corresponding output heat maps were given. From these heat maps, it was seen that the trained deep CNN model could learn the dental carries in the given panoramic dental images. The red regions in the heat maps showed the dental carries that the proposed model has learned. VOLUME 10, 2022

A. EFFECT OF PRE-TRAINED PARAMETERS
Learned weights of a pre-trained AlexNet architecture were used in the proposed model. In this study, the pre-trained deep model has been used because it contains parameters trained by using an extensive multi-class data set [38]. In this study, a deep architecture based on the transfer learning approach was used and adapted to our work. The training from the scratch model was used for dental caries' classification to test the effect of this process. More precisely, the process started with random weights instead of trained parameters of the deep architecture. Accuracy scores obtained from these experimental studies are given in Table 5.
As seen from Table 5, the proposed model achieved the best accuracy score with 92.17%. Additionally, raw images based on AlexNet and filtered images based on AlexNet reached mean accuracy scores of 86.35% and 86.25%, respectively. According to all these results, the AlexNet model, which has been based on the transfer learning approach and used for the proposed model, provided approximately 10% more success than the Training from scratch approach. Besides, this performance rise was observed for other methods. Based on these results, it has been proven that using trained parameters based on pre-trained deep models for classification problems with a small data set provides a significant performance increase.

B. COMPARISON OF THE STATE-OF-ART
There are many Deep convolutional Neural Network models developed for object recognition and classification in the literature. These architectures have been trained using a vast multi-class dataset. In this study, the proposed model was compared to seven pre-trained deep models: VGG16, VGG19, SqueezeNet, GoogleNet, ResNet18, ShuffleNet Xception, MobileNetv2, DarkNet19, and Inceptionv3. The same deep parameters were used for these architectures and the proposed model (Table 3). Besides, the same training/testing datasets were used, and experimental studies were repeated 100 times for these test data. Minimum, Maximum, Mean, and Standard deviation values for these results are given in Table 6.
In Table 6, the performance results of the pre-trained deep model are given. According to these results, the highest mean accuracy score among deep models achieved was 93.40% with the DarkNet19 model, the worst mean accuracy score was 80.25% with the Xception model. The highest maximum accuracy score was 97.39% using VGG16, SqueezeNet, and DarkNet19 models. As understood from Table 6, the proposed model to detect dental caries has achieved higher performance than pre-trained deep models.

V. DISCUSSION
Many studies have been carried out to detect dental caries. Traditional machine learning and deep learning methods have been used in these existing studies. Besides, Periapical X-Ray images have been often used to test these methods used in previous studies. The details and accuracy scores of these VOLUME 10, 2022 studies are given in Table 7. In the previous studies given in Table 7, different datasets were used.
As seen from Table 7, the proposed model produced a better accuracy score than the compared existing studies for dental caries detection. While Geetha et al. [5] among previous studies achieved the best accuracy level as 97.1%, Singh and Sehgal [7] produced the worst accuracy score with 86%. Furthermore, existing studies have produced an average accuracy score of 90%. As a result, the current study is one of the most comprehensive experimental studies conducted for dental caries detection, considering the previous research. Additionally, the proposed model having performance close to 100% can assist dentists in real-time computer-aided systems.

VI. CONCLUSION
With the developing technology, dental imaging tools improve day by day. These tools are highly efficient in supporting the professional skills of dentists. For the last decade, artificial intelligence techniques have been being used with dental imaging tools for developing decision support systems for dentists. This study proposes a novel decision support system for the automatic diagnosis of dental caries based on periapical images. The proposed approach is based on a multi-input deep convolutional neural network ensemble model (MI-DCNNE). A periapical image dataset covering 340 both caries and non-caries images has been used for performance evaluation of the proposed method. According to the obtained results, it is seen that the proposed model is quite successful in the diagnosis of dental caries, where the reported accuracy score is 99.13%.
In future works, the statistical analysis of the results from the various deep CNN models is investigated to determine how reliable the reported differences in mean accuracy scores.