LIME-Assisted Automatic Target Recognition With SAR Images: Toward Incremental Learning and Explainability

Integrating an automatic target recognition (ATR) system into real-world applications presents a challenge as it may frequently encounter new samples from unseen classes. To overcome this challenge, it is necessary to adopt incremental learning, which enables the continuous acquisition of new knowledge while retaining previous knowledge. This article introduces a novel, multipurpose interpretability metric for ATR systems that employs synthetic aperture radar images. The metric leverages the local interpretable model-agnostic explanation algorithm, enhancing human decision-making by providing a secondary measure alongside the conventional classification score. In addition, the proposed metric is employed to analyze the robustness of convolutional neural networks by examining the impact of target features and irrelevant background correlations on recognition results. Finally, we demonstrate the effectiveness of the proposed metric in the context of incremental learning. By utilizing the proposed interpretability metric, we select exemplars in an incremental learning scenario, resulting in improved performance and showcasing the application potential of our proposed methodology. The network is fine-tuned sequentially with unknown samples recognized by the Openmax classifier and exemplars from the old known classes, which are selected based on the proposed interpretability metric. The effectiveness of this approach is demonstrated using the publicly available MSTAR dataset.


I. INTRODUCTION
S YNTHETIC aperture radar (SAR) is a highly versatile technology used for a wide range of civilian and military applications, thanks to its exceptional ability to produce highresolution images of wide areas in all weather conditions [1].In radar surveillance and reconnaissance, automatic target recognition (ATR) is an important and challenging application of SAR images.Conventional supervised neural network approaches require large labeled datasets, which are scarce in SAR-ATR applications, and the process of precise object detection and labeling can be costly.Furthermore, it is impractical to gather every possible input in real-world scenarios [2].Note that the system might encounter new targets that were not included in previous training phases.In the first step, open-set recognition (OSR) techniques can help the system recognize unknown images.Subsequently, the network can be updated with the addition of new samples and classes [3].The conventional method of updating the network involves batch processing of all previous and new samples, performed offline.Instead, incremental learning techniques attempt to simulate human cognitive processes by continuously learning new tasks and adapting the current network, rather than undergoing a complete retraining process [4].
A common issue in network fine-tuning is an inclination toward the new classes.The balance between stability and plasticity, i.e., the ability to retain knowledge from previous data (stability) and the ability to acquire knowledge from new data (plasticity), is often referred to as the "stability-plasticity dilemma" [5].The fine-tuning process can cause catastrophic forgetting in the network, as the weights previously optimized for the classification of old classes are altered to handle the new class, resulting in the network losing its previous knowledge.Fig. 1 illustrates the catastrophic forgetting problem.In this specific example, the network has been trained to classify trucks and bulldozers, and it is successful in the upper portion of Fig. 1 when classifying a truck.However, once the model is fine-tuned to learn a new class, such as tanks, it fails to retain its earlier knowledge and mistakenly labels the same truck image as a bulldozer [6].This issue is referred to as catastrophic forgetting and occurs when there is a difference between the distributions of previous and new classes.
To tackle this challenge, "data replay" is a widely used approach in incremental learning [7].It involves retraining the network through exemplar sets to preserve accuracy in previous classes.Exemplar sets are small, representative collections of old data to replace complete retraining.Exemplar selection plays a crucial role in numerous machine learning (ML) applications and it can rely on performance metrics, such as classification Fig. 1.Catastrophic forgetting problem [6].
confidence [8], statistical distributions like proximity to the class mean [9], or even random sampling [10].For low-risk tolerance applications like SAR-ATR, where it is required to maintain a high level of accountability and transparency, interpretability can be used to select exemplars.Reliable and stable classification can only be achieved with a model that pays attention to target features, and not to spurious correlations from background clutter.The dependence on false correlations is called the "Clever Hans."Models that rely on these correlations tend to fail when they are applied in real-world scenarios where such correlations may not exist [11].The Clever Hans phenomenon is named after a horse in the early 1900s that appeared to perform arithmetic but was later discovered to be reading the examiner's body language [12].When processing images, the human vision system can quickly search, locate, and track the area of interest, while ignoring the background areas.Inspired by this mechanism, researchers introduced the concept of saliency detection to estimate the probability of each region in a given image to attract people's attention [13].
The advancement of deep learning (DL) models, known for being the least transparent among all ML methods, has sparked significant interest in interpretability and explainability [14], [15], [16], [17], [18], [19].Despite their remarkable performance, deep neural networks (DNNs) are frequently viewed as "black boxes" due to their lack of interpretability.There is no agreed-upon mathematical definition for explainable artificial intelligence (XAI) in ML, but a model is typically regarded as interpretable if the rationale behind the model's decision, i.e., the relationship between features and decisions can be comprehended.Note that XAI is a new field of research, and there is no universally accepted standard for its terminology.Some studies link the concepts of "interpretability" and "explainability" to notions, such as "reliability," "trustworthiness," and other similar terms [15].In addition, there is no clear consensus on how to distinguish between the terms "explainability" and "interpretability."Some studies, such as [15], use them interchangeably, while others suggest some minor differences [16].According to Arrieta et al. [14], interpretability is a passive characteristic of a model that shows how well a model makes sense for a human observer, while explainability is an active characteristic of a model referring to any procedure taken by a model to justify its internal functions.It is imperative to point out that as the number of features grows, even linear models become intricate and challenging to interpret, exacerbating the interpretability issue in DNNs that possess millions of parameters.Nevertheless, linear models are generally considered self-explanatory due to their straightforward internal mechanisms.An influential work in the field of XAI is the local interpretable model-agnostic explanation (LIME) method [20], which can be utilized for the visual interpretation of convolutional neural networks (CNNs) and other models.
This article introduces a new multipurpose interpretability metric based on the LIME algorithm.The primary use of the proposed metric is to assist the human operator in the decisionmaking process for ATR by offering a secondary performance measurement in addition to the standard classification score.In addition, this metric is used to measure how robustly the CNN performs overall by analyzing the ratio of results that are based on target features compared to those based on irrelevant correlations from background clutter.This assessment provides an indication of the CNN's ability to reliably detect target features while disregarding irrelevant patterns from the background clutter, hence determining its robustness.
In our prior study [6], we have examined the utilization of random sampling for exemplar selection with SAR images.In the present work, the proposed interpretability metric is integrated into an exemplar selection process to enhance the performance of incremental learning in SAR-ATR.The incremental learning process is combined with the concept of OSR using the wellknown Openmax classifier [21].The network is fine-tuned with both old class exemplars and new samples whenever a new class is recognized by Openmax.In this study, the publicly available moving and stationary target acquisition and recognition (MSTAR) dataset [22] is used in real data analysis to verify the theoretical findings.The main contributions of this study are threefold as follows.
1) Aiding human operators in ATR decision-making by providing an interpretability metric as a secondary performance indicator alongside the conventional classification score, to increase the operator's ability to assess the classifier's validity.2) Examining the statistical robustness of CNNs using the proposed interpretability metric to determine the extent to which results are based on relevant target features versus unimportant correlations from the background clutter.3) Improving the performance of incremental learning scenarios in SAR-ATR using the proposed LIME-assisted exemplar selection scheme.The rest of this organized as follows as follows: Section II provides an overview of current research in the fields of XAI and incremental learning using SAR images.Section III details the methods and materials used in the study, including the LIME algorithm for interpretability and the Openmax classifier for open set recognition.The proposed workflow is thoroughly explained in Section IV.The real data analysis is demonstrated in Section V. Section VI provides a concise discussion of the findings.Finally, Section VII concludes this article.

A. XAI in the SAR Domain
This subsection provides an overview of the current stateof-the-art methods in XAI specifically based on SAR images.In order to facilitate a deeper understanding of XAI, a comprehensive taxonomy that elucidates various aspects of XAI is presented in Fig. 2. XAI methods can be broadly categorized into two groups-ante-hoc and post-hoc methods-depending on how the explainability is achieved, either through simple models or by leveraging techniques to unveil complex trained models.A self-explainable ML model is typically referred to as ante-hoc, transparent, or intrinsic.Some notable ante-hoc XAI models include linear regression, logistic regression, decision trees, and rule-based learners.In contrast, post-hoc XAI refers to explainability methods that do not interfere with the training process and focus on interpreting already-trained, complex models without necessarily creating interpretable models [23].Post-hoc XAI can be further categorized into model-specific and modelagnostic methods.A model-specific XAI method is designed for a specific ML model, such as CNN, support vector machine (SVM), multilayer perceptron (MLP), recurrent neural network (RNN), and so on.One seminal example of a model-specific XAI method for CNNs is the class activation map (CAM) [24], which generates a heatmap showcasing areas of interest to the model.CAM and its variants, including gradient-weighted CAM (Grad-CAM) [25], score-CAM [26], ablation-CAM [27], etc., use weighted summation of final feature maps.In contrast, model-agnostic XAI methods can be applied to any ML model and treat them like a black box without accessing their internal mechanisms.For instance, LIME, which was mentioned earlier, is one of the most popular model-agnostic XAI methods.The game-theoretic-based approach, namely, SHapley Additive ex-Planations (SHAP) [28], is another very popular model-agnostic XAI method.It is worth mentioning that ante-hoc XAI methods are always, by definition, model-specific.While devising a comprehensive taxonomy for XAI methods is very challenging, many scholars have attempted to categorize them based on different aspects.For example, some studies divide XAI methods into global and local explainability scopes [14].Global approaches explain an ML model as a whole, i.e., by trying to identify common important patterns across all predictions, whereas local approaches aim to explain an ML model using a single prediction.
In what follows, we address various state-of-the-art XAI methods in the SAR-ATR field.Heiligers et al. [29] utilized the LIME algorithm, while Pannu et al. [30] employed the Grad-CAM algorithm to visualize CNN decision-making processes on MSTAR images.Panati et al. [31] conducted a comparative analysis of LIME, Grad-CAM, and SHAP to evaluate feature relevance in the MSTAR dataset.Zang et al. [32] used layerwise relevance propagation (LRP) to interpret CNNs on the MSTAR dataset.Belloni et al. [33] introduced a method where a small black kernel was slid over MSTAR images, the percentage of correctly classified images was measured, and a classification map was created based on the new intensity.This analysis allowed them to study the individual contributions of targets, shadows, and backgrounds in SAR-ATR.Although the target's shadow area contains essential information, simultaneous utilization of both the target and shadow features is challenging.Shadow regions typically have significantly lower intensity levels compared to the target or clutter regions.As a result, their semantic signatures are prone to suppression during max-pool-based feature encoding in general DNN models [34].Zhu et al. [35] proposed a data selection method based on LIME to improve the quality of images generated by a generative adversarial network (GAN).This method used only the selected set of SAR images, where LIME positively coincided with targets, preventing the GAN from making "Clever Hans" decisions.However, this approach has the limitation of being a manual data selection process.Feng et al. [36] introduced the self-matching CAM, a new heatmap generation method to provide visual explanations for CNNs.By comparing it to other state-of-the-art CAM methods, they demonstrated that self-matching CAM more accurately highlights target regions in SAR images.In [37], the same authors proposed a less computationally demanding version of their former work, named spectral clustering self-matching CAM, which eliminated the need for performing hundreds of self-matching operations per image.Li et al. [38], [39] proposed an ante-hoc interpretable recognition model for the MSTAR dataset based on the interpretability of bag-of-feature (BoF) models.This model was inspired by BagNets [40], which classifies images according to the count of local features rather than their spatial relationships.

B. Incremental Learning in the SAR Domain
In the context of incremental learning using the MSTAR dataset, several studies have been conducted to address key challenges, such as model degradation and catastrophic forgetting.Tang et al. [5] proposed a framework that employs multiple models trained on previous tasks to correct cumulative errors and combat plasticity decline.Zhou et al. [2] employed the concept of knowledge distillation, initially used in transfer learning, to address the issue of catastrophic forgetting.Dang et al. [4] proposed an incremental nonnegative matrix factorization (INMF) with sparse constraints, which is a dynamic feature learning method to update the trained model incrementally, considering the sparse characteristics of scattering centers in SAR images.In addition, the same authors proposed an exemplar selection technique [3], which identifies the edge exemplars of each class and fits probabilistic distributions to them.To ensure that previous class boundaries are not broken when new classes are added, they also introduced a class boundary sample synthesis method for incremental learning [41].Liu et al. [42] proposed an incremental multitasking SAR target recognition method to simultaneously recognize both the class and aspect angle of SAR targets.Their method is based on structured pruning to identify the dominant neuron.Huang et al. [43] proposed an approach termed memory augmented weights alignment and enhancement discrimination, incorporating the attention mechanism, to tackle catastrophic forgetting in SAR incremental learning scenarios.Wang et al. [44] tackled the challenge of few-shot learning in the context of SAR incremental target recognition.They addressed the continuous recognition of new target classes despite having only limited training samples.Lu et al. [7] proposed a lightweight incremental learning approach for scene classification in remote sensing images.Their approach uses a dual-constraint loss that integrates knowledge distillation and adversarial training to supervise the training of the feature transfer module at both the feature and semantic levels.Liu et al. [45] proposed a controllable convex hull-based exemplar selection strategy to alleviate the catastrophic forgetting problem in incremental learning for scene classification in remote sensing images.Nie et al. [46] addressed the problem of online incremental learning for multiview classification in polarimetric SAR (PolSAR) data.Their method integrates both the multiview coregularization and graph regularization techniques, leveraging the disagreement among multiview predictors.Fan et al. [47] proposed an incremental Wishart learning approach for PolSAR image classification by transferring Wishart distribution, other polarimetric decomposition, and spatial features to establish a one-layer structure.

A. LIME for the Explainability
A high-level schematic of the LIME algorithm is presented in Fig. 3.The initial step of LIME involves generating multiple perturbed instances from the input sample x, which are subsequently utilized to train a simple, interpretable local surrogate model.It is worth noting that the dimensionality of the interpretable model does not have to match that of the black-box model.Moreover, in the case of image inputs, perturbing individual pixels is not a meaningful approach [48].Therefore, the input image needs to be oversegmented into superpixels using well-established methods, such as the Quick-Shift [49], Slic [50], Felzenszwalb [51], etc.A "superpixel," which represents a feature, refers to a group of adjacent pixels that share similar visual properties.Perturbed instances can then be created by turning OFF some superpixels in each permutation, similar to the occlusion sensitivity technique [52], by setting the pixel values to a constant.Note that selecting a larger number of superpixels leads to a greater number of smaller superpixels, whereas a lower number of superpixels results in fewer but larger superpixels.Thus, in our particular ATR case study, choosing the appropriate number of superpixels depends on the target size and the desired level of segmentation.
Let us consider z , a perturbed instance in the interpretable domain, with z representing its correspondence in the image domain.For example, a random binary vector with N s elements, z = {0, 1, 1, . . ., 0}, can be regarded a perturbed instance in Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
the interpretable domain, where N s denotes to the number of superpixels.To translate from the interpretable domain (z ) to the image domain (z), pixels within the superpixels designated as "0" should be set to a constant value, while other superpixels should remain unchanged.The perturbed image z is subsequently fed into the black-box model F, and its output prediction f(z) is stored.By repeating this process, a new dataset {z , f(z)} can be created, consisting of the perturbed instances z and their corresponding labels f(z), to train the surrogate model.An interpretable model is then selected as the surrogate model for locally approximating the black-box model in the vicinity of the input sample.This surrogate model is trained using the perturbed instances, which are weighted based on their proximity to the input sample.These weights ensure that the surrogate model is locally faithful, with higher weights assigned to perturbed instances that are more similar, i.e., have lower distance, to the input sample.Mathematically, LIME aims to find the local surrogate model (g) that explains the black-box model (f) in the vicinity (π x ) of the input (x) by where η(x) refers to the explanation of x, and G refers to a set of interpretable models.Apart from minimizing the loss function L, LIME also ensures that the surrogate model remains simple by incorporating the penalty term Ω, which represents the complexity of the surrogate model.To weigh a perturbed image z based on its similarity to the input sample x, LIME utilizes the Gaussian radial basis function (RBF) kernel where σ represents the width of the kernel.As previously stated π x assigns greater weight to instances that are less perturbed compared to the input x, with its value ranging between 0 and 1.Subsequently, the locality-aware loss L(f, g, π x ) can be approximated by weighted least squares utilizing the perturbed instances that have been weighted by Assuming the class of linear models, the surrogate model g can be defined as where ω ω ω = ω 1 , . . ., ω N represents the explanation provided by LIME.The applicable penalty term for the complexity of the linear regression model can be given by where .0 denotes the l 0 norm, which indicates the number of nonzero elements in ω ω ω.Note that T p can be considered as a tuning parameter that regulates the complexity of the surrogate model g.By combining the preceding formulas ( 2)-( 5), we can reexpress (1) for the linear regression case as where M denotes the number of generated perturbed instances, and z i and z i represent the perturbed instances in the interpretable and image domains, respectively.It is noteworthy to recall that LIME provides a local approximation, which means that for each input x, several perturbed instances need to be regenerated and the surrogate model g should be retrained.Moreover, the size of the dataset is not important.

B. Openmax for Open Set Recognition
The Openmax approach [21], which is a key component of our proposed methodology, is presented succinctly in this subsection.Openmax consists of two internal phases, which are as follows.The initial phase extracts statistical features from known data, while the subsequent phase introduces a new class to the activation vector (v) of the test image, enabling unknown recognition.According to [21], the activation vector v refers to the output of the final fully connected layer in the CNN.The first phase starts with the calculation of the mean activation vector (μ μ μ) for each training class where N denotes the number of training classes, the activation vector of the jth image in class k is denoted by v v v k j ∈ R N ×1 , and the mean activation vector of class k, represented by μ μ μ k ∈ R N ×1 , is calculated using N k images in class k.The next step is to calculate the distance between the activation vectors of images and their corresponding mean activation vector in each class where is the distance scalar between the jth image in class k and the mean activation vector of class k, and . 2 represents the l 2 norm.Subsequently, D k is processed and its largest η values are selected.The hyperparameter η, which is a scalar value also referred to as the "tail size," is used for this selection.The selected values are used to fit a Weibull distribution, and its cumulative distribution function (CDF) is computed as where FitHigh(.) is a function, provided by the open-source meta-recognition library LibMR [53], to perform the Weibull fitting on the η largest values of D k .Note that τ k , κ k , and λ k refer to the shift, shape, and scale, respectively, of the estimated Weibull distribution for the training class k, respectively.In the second phase, Openmax modifies the activation vector of the test image by adding a new element at the end to stand for the and the new class to represent unknown targets is appended as Note that the weights ω in (11) only scale the top α elements of the activation vector by where α is another hyperparameter of Openmax, defined as the number of "top" classes to revise, and s = argsort(v) sorts the activation vector to provide the indexes in descending order.Openmax scores are then calculated using (11) and (12) by In the end, the test image is recognized as unknown if In our previous studies [54], [55], [56], [57], we have investigated the application of Openmax in various open set scenarios using SAR images and the potential challenges it may encounter.

A. Interpretability Metric
As previously stated, the primary motivation to derive the interpretability metric is to assist human operators in decisionmaking by providing a supplementary performance criterion, in addition to the standard classification score generated by the Softmax module in a CNN.This allows the operator to determine if the CNN is focusing on the important features of the target and not just any irrelevant correlations in the background.In addition, by calculating the histograms of the proposed interpretability metric for each class, it is possible to examine the robustness of the CNN against Clever Hans decision errors.This helps in recognizing any particular class or classes that are more susceptible to relying on background clutter and in devising measures to enhance the model's robustness against such dependencies.The proposed metric, whose derivation process is shown in Fig. 4, employs the LIME algorithm to identify the main superpixel, D LIME .In the second branch of Fig. 4, the input image is thresholded at the pixel level, then postprocessed to extract the target area, referred to as D Target .The proposed interpretability metric ω is calculated as follows: where and  identical, the interpretability metric attains its maximum value of ω = (1 + 1)/2 = 1.On the other hand, if the two areas do no overlap, D LIME ∩ D Target = ∅, the interpretability metric reaches its lowest value, ω = 0.The interpretability metric is undefined and referred to as NaN (not a number) if one of the denominators is zero.For example, an incorrect threshold or poor postprocessing may result in an empty D Target , leading to ω 1 = NaN and an undefined interpretability metric.The postprocessing module, which is designed to detect and remove outliers from the input image, is described in the following.Let us define I Threshold , the binary image resulting from the comparison between the input image I and the threshold T as the input of the postprocessing algorithm.This algorithm is comprised of two internal functions, called Outlier Removal(.)and Gap Filler(.), as shown in Fig. 5.The Outlier Removal(.)function aims at removing sparse outliers, while the Gap Filler(.) function interpolates missing parts in the main target.This process is repeated either for a specified number of iterations (N iter ) or until the difference between the input and output is less than a predefined constant (δ min ).To better understand how Outlier Removal(.)works, with reference to Fig. 6, a small square kernel with a side length of 2R 1 +1 is slid over the binary image, I Threshold .If the sum of surrounding Algorithm 1: Pseudocode for the Postprocessing Algorithm.
pixels is less than a threshold T 1 , the value of the central pixel will be replaced with zero.Conversely, Gap Filler(.) sets the value of the central pixel to one if the sum of surrounding pixels is greater than a threshold T 2 .To increase generalizability, a different kernel size of 2R 2 +1 is used for Gap Filler(.).The values of T, together with R 1 , R 2 , T 1 , and T 2 are all set empirically through a process of hyperparameter tuning.To provide a concise explanation of the postprocessing algorithm, we present a pseudocode in Algorithm 1, corresponding to Fig. 5.The algorithm takes the binary image I Threshold , as input and generates the processed image, I Threshold,Processed .To enhance clarity, we encapsulate Outlier Removal(.)and Gap Filler(.) into a new function named Process Module(.).The loop continues until either N iter is reached or the difference (δ) between the output of Process Module(.) and the binary image (I Threshold ) is less than δ min .The difference between the two images is measured by the Frobenius norm ( .F ), and δ min = 0 indicates a perfect match between the output of Process Module(.) and I Threshold .
The pseudocode for Process Module(.) and its two internal functions, Outlier Removal(.)and Gap Filler(.), are presented in Algorithms 2-4, respectively.Note that N row and N column refer to the number of pixels in the input image in the vertical and horizontal directions, respectively.

B. Incremental Learning
Another motivation to derive the interpretability metric is to select exemplars in an incremental learning scenario with SAR images.The framework for incremental learning is depicted in Fig. 7, where the Openmax classifier is incorporated as a  class is introduced in the test phase after each fine-tuning.To prevent imbalanced classes, we propose LIME-assisted exemplar selection in the old classes.This mitigates the likelihood of the network becoming biased toward the old classes.The LIME-assisted exemplar selection process begins by calculating the interpretability metric for all images that belong to old classes.Based on this analysis, a specific number of images with higher interpretability scores are selected from each class to create smaller, well-balanced, and more meaningful representative sets.The primary objective of exemplar selection is to enhance the efficiency of data analysis by reducing the number of samples while ensuring the representativeness of the data.It can be particularly beneficial for managing large datasets or when computational time and resources are constrained.The LIME-assisted exemplar selection procedure, in contrast to random sampling, selects the most informative samples, which Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.gains a deeper understanding of the data and improves the accuracy, particularly in incremental scenarios.We acknowledge that ω 1 and ω 2 in ( 16) hold different meanings, and relying on the average value might not be the optimal approach.However, in the context of exemplar selection for incremental learning, it is more practical to work with a single reliability indicator.To establish a single reliability measure that still accounts for the differing impacts of ω 1 and ω 2 , one could consider forming their weighted sum as ω = αω 1 + (1 − α)ω 2 , where α represents a balancing parameter between 0 and 1.Nonetheless, this approach would introduce further complexities to the case studies.The proposed incremental learning framework is henceforth referred to as LIME-ITR, which stands for LIME-assisted incremental target recognition.It is important to note that Openmax is not activated every time a new image is received.Instead, the framework must receive a batch of several images before calling the Openmax method to minimize unnecessary computational overhead.

V. REAL-DATA ANALYSIS
In this article, we utilize the public MSTAR dataset, which is released by the U.S. Air Force Research Laboratory and the Defense Advanced Research Projects Agency.The dataset comprises X-band SAR images of ten distinct classes of groundbased military targets.In Fig. 8, SAR and optical images of these ten target types in the dataset are shown.The numbers within parentheses indicate the number of images captured at depression angles of 17 • and 15 • , respectively.According to standard operating conditions (SOC), the train and test sets are typically segregated by depression angles of 17

A. Credible Classification
In our first experiment, we evaluate the reliability of the classification results using the proposed interpretability metric.To simplify the analysis, we train a CNN using only three classes in this subsection.The CNN's architecture is outlined in Table I, with the first two convolutional layers (Conv 1 and Conv 2) having a kernel size of 3 × 3, while the last one has a kernel size of 5 × 5.All pooling layers employ a 2 × 2 kernel and two fully connected layers (Dense 1 and Dense 2) are utilized to learn nonlinear combinations of previous high-level features [58].Note that in Table I, the activation function is referred to as "Act.Func.," with rectified linear unit (ReLU) applied to all convolutional filters and Softmax applied to the last fully connected layer.Furthermore, "Param" in Table I represents the number of trainable parameters.The CNN is implemented using the Keras library backend to TensorFlow 1.14.0 with Python 3.6.13.The cross-entropy loss function is minimized using the Adam optimization algorithm with a learning rate of 10 −3 .All experiments were conducted on a personal computer with a 12th Gen Intel(R) Core(TM) i7-12700 processor, Nvidia GeForce GT 1030 GPU, and 64 GB RAM.
The performance metrics, including precision, recall, F1score, and accuracy, obtained by evaluating the CNN using the test dataset, are presented in Fig. 9 along with the confusion matrix (CM).As illustrated in Fig. 9, the CNN model classifies most of the test images with high accuracy.Note that "support" Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
in the last column in Fig. 9(b) indicates the number of test images in the corresponding class.Now we calculate the proposed interpretability metric to determine how the CNN has gained relevant and irrelevant knowledge from the training dataset.In Fig. 10, we present three different test examples.In Fig. 10(a), a test example from class 0 is depicted, where the LIME outputs and the results after applying a threshold and postprocessing algorithm are shown in the first and second rows, respectively.The Softmax classifier achieves a correct classification score of 99%, and the interpretability metric validates this result with a value of 0.785.In Fig. 10(b), a test example from class 1 is presented, where Softmax still has a high confidence score of 99%; nevertheless, the interpretability metric is 0, indicating an unreliable classification.Lastly, Fig. 10(c) shows a test image from class 2, with a confident Softmax classification score of 98%.The interpretability index is 0.29, which confirms the Softmax classification at a lower level compared to Fig. 10(a).In the previous case studies, the values of R1 and R2 were set to one, while T1 and T2 were equal to four.
The following analysis focuses on one of the five images that were misclassified by the Softmax classifier, presenting a challenging case.As illustrated in Fig. 11, for a test example belonging to class 0, Softmax yields a relatively high incorrect classification score of 99% in favor of class 2. Despite this, the interpretability metric ω = 0.615 indicates a reasonable degree of interpretability.In such a scenario, a human operator may erroneously rely on the CNN's classification outcome, as both metrics appear to be sufficiently high.This problem can be addressed through the lens of robustness.
Note that the above test samples are solely utilized to illustrate the methodology behind the proposed interpretability metric.In fact, this small set of examples does not suffice to thoroughly analyze the robustness of the CNN, i.e., its ability to deliver satisfactory performance in alternative clutter conditions.To address this, the histograms of ω 1 , ω 2 , and ω for the three classes are depicted in Fig. 12.It is evident that the CNN has effectively learned the key features of classes 0 and 2, but it seems to have utilized irrelevant features for class 1.Nonetheless, the accuracy of the classifications is quite high, with a mere five misclassified images out of a total of 744 test images.
Fig. 13 shows a few more examples to illustrate how this model mostly uses the background information in class 1, whereas in other classes, the focus is on the target.For the sake of conciseness, we have only displayed LIME's outputs, where the main superpixel is highlighted in green.One plausible approach to refocus the model's attention from the background to targets in class 1 would be to modify the model's hyperparameters or to alter its architecture.
Regarding computational time, training the CNN with the dataset took an average of 46 s for 15 epochs, while a single prediction required only 4.5 ms.For LIME processing of one image with one feature (superpixel) and 1000 perturbed data samples, the average time was 1.45 s.As for the postprocessing algorithm with N iter =10, it required an average of 206 ms to complete.

B. Exemplar Selection: A Two-Step Incremental Learning Scenario
In this subsection, we conduct an analysis of a two-step incremental learning scenario to detail the various steps of the proposed method.The secondary objective of this analysis is to present the effectiveness of our proposed LIME-assisted scheme in comparison to other exemplar selection methods within an incremental learning scenario.In other words, in this subsection, we evaluate our proposed Openmax-based incremental learning framework using different exemplar selection methods: random sampling, incremental classifier and representation learning (iCaRL) [9], class boundary selection-based incremental learning (CBesIL) [41], and the proposed LIME-assisted sampling.iCaRL [9] selects a certain number of exemplars for each class, based on memory size, and utilizes two routines for exemplar management, i.e., selecting exemplars for new classes and reducing the size of the exemplar sets of previous classes.iCaRL selects exemplars that are closer to the average feature vector of training data and discards the least important exemplars based on their position in the prioritized list.On the other hand, CBesIL [41] divides the feature space into overlap, edge, and interior regions.It selects exemplars from the overlap region first, which carries more classification information and is the primary source of errors.The edge region is important for detecting novel classes, and its exemplars help form the decision surface.The algorithm identifies the k 0 nearest neighbors for each training example and determines the number of classes in those neighbors.Based on the highest Bayes posterior probability the boundary exemplar sets are formed.
The simulation scenario is outlined in Fig. 14 in which the CNN structure presented in Table I is utilized.Initially, the model is trained on four classes in Set 1 = {2S1, BRDM2, D7, ZiL131}.The number of classes in this scenario has been limited to simplify the understanding of how the proposed approach can be applied to the task of incremental learning.Note that in each round of fine-tuning, the number of elements in the last fully connected layer is adjusted to match the number of classes.The CM, integrated with performance indicators, is illustrated in Fig. 15(a), demonstrating that the Softmax classifier accurately classifies the majority of test images from the four classes as expected.In addition, Fig. 15 exhibits eight confusion matrixes corresponding to eight classifiers (a)-(h) of random and LIME-assisted sampling methods in Fig. 14.For the sake of conciseness, we have only shown the CMs of these two methods.Note that, in contrast to Fig. 9(a), the CMs shown in Fig. 15 have not been normalized to highlight variations in the number of test images among all classes.In the following part, a detailed description of the incremental learning scenario depicted in Figs. 14 and 15 is provided.
In the first step of the incremental learning scenario, a new class, ZSU23/4, is introduced to the network.This class, which is indicated by a red arrow in Fig. 14, has not been included in the initial training phase.The CM of the Openmax classifier is depicted in Fig. 15(b), which shows that Openmax has effectively rejected 48.5% of the total 274 unknown images.A new label, "4," is assigned to the rejected images, which, along with exemplars from the previous four classes, form the new training set (Set 2).To accommodate the new class, the number of elements in the final layer of the CNN is increased from four to five, and the network is fine-tuned using Set 2. The overall accuracy of Softmax classifier for different exemplar selection methods are summarized in Fig. 14 (1st step of incremental learning).Moreover, the CMs for the networks using random sampling and LIME-assisted exemplar selection are depicted in Fig. 15(c) and (d), respectively.All exemplar selection methods successfully learn the new class and retain their knowledge of the previous classes.However, the results show that the LIMEassisted approach outperforms random sampling, iCaRL [9], and CBesIL [41], methods with an overall accuracy of 88%, compared to 83.8%, 84.5%, and 86.8%, respectively.
In the second step of the incremental learning scenario, another new class (T62) is introduced to the networks.The CM shown in Fig. 15(f) indicates that Openmax has successfully rejected 58.2% of the total 273 unknown images in the LIMEassisted sampling approach.Following the same process as in the first step, the rejected images are assigned a new label ("5"), and a new training set (Set 3) is generated by combining the new class and the exemplars from the preceding five classes.The final layer of the CNN is expanded from five to six elements, and the network is retrained using Set 3. The overall accuracies of the Softmax classifiers under different exemplar selection methods are shown in Fig. 14 (2nd step of incremental learning).The LIME-assisted approach achieves an overall accuracy of 87%, whereas the random sampling, iCaRL [9], and CBesIL [41], have overall accuracies of 78.2%, 78.4%, and 84.9 %, respectively.In addition, the CMs of the Softmax classifier for the random-sampling-based CNN and the LIME-assisted approach are demonstrated in Fig. 15(g) and (h), respectively.It is clear that the LIME-assisted network effectively learns the new target T62, while also retaining knowledge of the previous classes.
In summary, the presented incremental learning scenario was initiated with four classes, ranging from 0 to 3, and through two incremental learning steps (S1 and S2), the network has learned two additional classes, namely, 4 and 5.The superiority of the LIME-assisted exemplar selection over other approaches is clearly illustrated in Fig. 16, which shows the recalls for each class.Note that "Init."refers to the initial training phase.The discrepancy between these methods is particularly noticeable in the last class.However, these methods are not comparable in terms of computational time because the LIME-assisted approach requires significantly more time for its internal calculations.After completing the initial training phase and Openmax process, which take 63.13 and 145.01 s, respectively, we narrow our focus to the first step of the incremental learning scenario to ensure a fair comparison.In order to select exemplars from the old classes, namely, Set 1 = {2S1, BRDM2, D7, ZiL131}, which consist of 1195 images, random sampling, iCaRL [9], CBesIL [41], and LIME-assisted, consume 0.03 ms, 2.2 s, 2.12 s, and 1704.27s, respectively.Note that the considerable computational time required by the LIME-assisted method is mainly attributed to the use of 1000 perturbed data samples, and it decreases approximately linearly to 181.91 s when using only 100 perturbed data samples.For the subsequent steps of the incremental learning scenario, while the time requirements of other methods do not change considerably, the time consumed by the LIME-assisted method is approximately linearly dependent on the number of images in the old classes, as the interpretability metric should be computed for each image separately.

C. Comparison With Previous Methods: A Six-Step Incremental Learning Scenario
In this subsection, we first extend the scenario discussed in the previous subsection, where the CNN is initially trained with four classes.Through six incremental learning steps (i.e., S1 to S6), the model learns all ten classes in the dataset.The classwise recognition accuracy of the proposed method through the six incremental learning steps is summarized in Table II.Note that the light gray elements represent the newest class that the network has learned, whereas the dark gray elements show the classes that the network has not yet encountered.This table provides a clear overview of the network's learning progress and its ability to adapt to new classes over time.The last column of Table II, shows the overall accuracy in each incremental learning step, and its variation from the previous step is indicated in parenthesis.The results from Table II indicate that the proposed method is able to quickly adapt to new data and maintain its accuracy for previously learned classes.
In what follows, a quantitative comparison of the proposed incremental learning method with two recent approaches, namely, Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.L p -INMF [4] and HPecIL [5], through a comprehensive scenario is provided.L p -INMF represents the incremental version of nonnegative matrix factorization (NMF) with L p sparse constraints.The main concept of NMF is to decompose a matrix (training matrix) into two multiplicative matrixes: 1) the base matrix and 2) the coding matrix.Incremental NMF (INMF) focuses on updating the trained model using only new samples.
The L p sparse constraint, where p typically ranges between 0 and 1, is introduced to the decomposition matrix during the update process to achieve higher recognition performance than the general INMF.On the other hand, HPecIL, which stands for high plastic error correction incremental learning, utilizes a loss function that combines classification loss and distillation loss to update model parameters in each incremental learning step.The    17. Comparison between the proposed method (LIME-ITR), HPecIL [5] and L p -INMF [4] in terms of overall accuracy through six incremental learning steps.
classification loss is computed using the stored exemplars from old classes as well as samples from new classes, whereas the distillation loss represents the error between predictions and soft targets, summed over previous models.These soft targets are the outputs of old model instances with the training data as inputs.HPecIL utilizes multiple previous models, rather than solely relying on the last model, to mitigate the accumulation of errors.
To provide a concise comparison between proposed method previous studies, Fig. 17 illustrates the overall accuracy of the proposed method (LIME-ITR) alongside those of L p -INMF [4] and HPecIL [5], plotted against the number of training classes.Overall accuracy provides a useful measure of the algorithm's effectiveness across all classes, rather than just individual ones.Note that we have set the sparse constraint p to 0.5, as it outperforms p = 1.Taking this scenario into consideration, L 1/2 -INMF [4] demonstrates lower accuracy across all incremental learning steps, in contrast to LIME-ITR and HPecL [5].In Fig. 17, it is observed that LIME-ITR experiences a larger drop in accuracy in the first two incremental steps compared to HPecIL [5] but surpasses HPecIL in subsequent steps.The 10% decrease in overall accuracy of Step 1 (S1) can be attributed to multiple factors.First, as detailed in Table II for further insights, this decline reflects the model's challenges in adapting to the new class, "ZSU23/4," which was not present in the initial training step.Second, it implies the presence of some level of forgetting or interference with the previously learned classes.Notably, the most noticeable decline in accuracy is observed in the class "D7," which drops from 99.3% in the initial step to 80.3% in Step 1 (S1).This occurrence of catastrophic forgetting implies that the model's adaptation to the new class (ZSU23/4) may have adversely affected its ability to accurately recognize the class (D7), which had been previously learned during the initial step.However, as the incremental learning scenario progresses and more classes are introduced, despite some fluctuations in accuracy, LIME-ITR consistently outperforms HPecIL [5], as demonstrated in the subsequent steps.This phenomenon can be attributed to LIME-ITR's adaptability to new classes while retaining knowledge of previously learned ones, ultimately resulting in superior overall accuracy in the subsequent steps of incremental learning scenario.It is important to note, however, that classwise recall may still vary between the methods.

VI. DISCUSSION
In this section, we examine both the advantages and potential areas for improvement of the proposed method in the context of the real-data analysis presented in the previous section.

A. Advantages
The key advantages of the proposed methodology are outlined as follows.
1) Credible Classification Using the Interpretability Metric: Through the analysis of the proposed LIME-based interpretability metric in conjunction with classification scores, the proposed method distinguishes between reliable and unreliable classifications.Furthermore, the histograms of the interpretability metric across different classes provide Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
valuable insights into the model's ability to extract relevant target features versus irrelevant background information.2) Incremental Learning Framework: In the context of exemplar selection, the LIME-assisted sampling method has proven its effectiveness in adapting to new classes while retaining knowledge of previously learned classes.This effectiveness stands out when compared to alternative exemplar selection techniques like random sampling, herding selection in iCaRL [9], and class boundary selection in CBesIL [41], as outlined in Fig. 16.The proposed method has also outperformed existing incremental learning approaches like L p -INMF [4] and HPecIL [5].This superiority is evident in the higher overall accuracy achieved by LIME-ITR across the incremental learning steps S3 to S6, as illustrated in Fig. 17.

B. Potential Improvements
While our proposed methodology shows promising results, there are potential areas for improvement as follows.
1) Interpretability Metric Refinement: Refining the metric's components and exploring its sensitivity to different model architectures could lead to a more reliable metric.2) Handling Challenging Cases: Developing strategies to handle the challenging cases, where the model's classification score and the interpretability metric are misaligned, could improve the overall reliability and robustness of the methodology.3) Generalization to Larger Datasets: Extending the experiments to larger and more diverse datasets would offer insights into the applicability of the proposed methods to different real-world scenarios.4) Hyperparameter Tuning: Investigating the optimal settings for hyperparameters, such as those in Openmax and LIME, might further enhance the overall performance of the method.5) Efficiency in Computational Time: Exploring methods to optimize the computational efficiency, specially concerning the internal calculations required by LIME, without compromising the accuracy, is worth considering.6) Robustness to Complex Backgrounds: Note that background of a target slice in MSTAR dataset is clean.Even in the presence of a complex background, it is still feasible to utilize the main superpixel identified by the LIME algorithm to determine the contributing portion of the image for classification.Nevertheless, improving the postprocessing algorithm to accurately distinguish the target's area from the complex background is necessary.As a potential solution, integrating an object detection algorithm like YOLO 3 [59] into the postprocessing step could significantly enhance the algorithm's performance.In conclusion, the presented methodology offers a number of advantages.However, there remains room for improvement, particularly in terms of improving robustness and accuracy.Addressing the aforementioned aspects has the potential to further expand its applicability across a variety of real-world scenarios.

VII. CONCLUSION
In this article, we ahve presented a multipurpose interpretability metric based on the LIME algorithm for ATR systems using SAR images.We have shown how the proposed interpretability metric can assist human decision-making by offering a complementary measure to the conventional classification score.
In addition, we have demonstrated the effectiveness of this metric in conducting statistical robustness analysis of CNNs and selecting exemplars in incremental learning scenarios.The present study highlights the importance of interpretability as a crucial factor in improving the performance and reliability of SAR-ATR systems in real-world applications.The results suggest that interpretability metrics can serve as a powerful tool for advancing the field of SAR-ATR and can be applied more broadly to gain a better understanding of the strengths and weaknesses of various ML models.Although our study has yielded promising results, it is important to acknowledge that the LIME algorithm has its limitations, such as sensitivity to hyperparameters, high computational cost, reliance on linear surrogate models, failure to comprehend global patterns, and lack of stability to small input variations.To address these limitations, future research should explore alternative XAI methods.In addition, a model-based augmentation technique [60] can greatly improve the adaptability of the model to new and unseen data, thereby enhancing its ability to generalize and make accurate predictions.In the end, the applicability of the proposed interpretability metric can be extended to ATR with 3-D polarimetric interferometry inverse SAR images, which can be reconstructed through 3-D point clouds [61], [62], [63], [64].
) are based on the intersection {∩} of D LIME and D Target .The symbol "|.|" indicates the number of pixels in the respective regions.The interpretability metric ω ranges from 0 to 1, excluding cases where the denominator is zero.If D LIME and D Target are Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

Algorithm 2 : 2 ) 3 :Algorithm 4 :
Process Module Algorithm.Inputs: I, R 1 , R 2 , T 1 , T 2 Outputs: I 2 Function: Gap Filler(.),Outlier Removal(.)1: I 1 ← Outlier Removal(I , R 1 , T 1 ) 2: I 2 ← Gap Filler(I 1 , R 2 , T Algorithm Outlier Removal.supplementary component of the CNN.As previously mentioned, the Openmax classifier assists the CNN in recognizing unknown targets during testing.The outputs of both the Openmax and Softmax classifiers are then processed together, and a new label is assigned to the unknown samples.These samples, constituting a new class, are subsequently included in the training dataset for the next fine-tuning stage.Furthermore, to maintain the cycle in the incremental learning scenario, a new Gap Filler.

Fig. 9 .
Fig. 9. (a) Confusion matrix and (b) performance indicators when the CNN is tested by three classes.

Fig. 13 .
Fig. 13.LIME's outputs are shown through a few examples: (a) class 0, (b) class 1, and (c) class 2. The main superpixel is highlighted in green.The CNN has learned target features of classes 0 and 2 but utilized background information in class 1.The final example in each row exhibits a distinct behavior.

Fig. 15 .
Fig. 15.Confusion matrixes and performance indicators for the incremental learning scenario: (a) Softmax for the initial state with four classes.(b) Openmax performance when a new target class is introduced.(c) Softmax using random sampling for the 1st round of incremental learning with five classes.(d) Softmax using LIME-assisted sampling for the 1st round of incremental learning with five classes.(e) Openmax using random sampling for the second new target class.(f) Openmax using LIME-assisted sampling for the second new target class.(g) Softmax using random sampling for the 2nd round of incremental learning with six classes.(h) Softmax using LIME-assisted sampling for the 2nd round of incremental learning with six classes.

Fig. 16 .
Fig. 16.Classwise recall for the incremental learning scenario: learning class 4 in the first incremental learning step, S1, and class 5 in S2.

TABLE II CLASSWISE
RECOGNITION ACCURACY OF THE PROPOSED METHOD THROUGH THE SIX INCREMENTAL LEARNING STEPSFig.