StrategyAtlas: Strategy Analysis for Machine Learning Interpretability

Businesses in high-risk environments have been reluctant to adopt modern machine learning approaches due to their complex and uninterpretable nature. Most current solutions provide local, instance-level explanations, but this is insufficient for understanding the model as a whole. In this work, we show that strategy clusters (i.e., groups of data instances that are treated distinctly by the model) can be used to understand the global behavior of a complex ML model. To support effective exploration and understanding of these clusters, we introduce StrategyAtlas, a system designed to analyze and explain model strategies. Furthermore, it supports multiple ways to utilize these strategies for simplifying and improving the reference model. In collaboration with a large insurance company, we present a use case in automatic insurance acceptance, and show how professional data scientists were enabled to understand a complex model and improve the production model based on these insights.


StrategyAtlas: Strategy Analysis for Machine
Learning Interpretability Dennis Collaris and Jarke J. van Wijk Abstract-Businesses in high-risk environments have been reluctant to adopt modern machine learning approaches due to their complex and uninterpretable nature.Most current solutions provide local, instance-level explanations, but this is insufficient for understanding the model as a whole.In this work, we show that strategy clusters (i.e., groups of data instances that are treated distinctly by the model) can be used to understand the global behavior of a complex ML model.To support effective exploration and understanding of these clusters, we introduce STRATEGYATLAS, a system designed to analyze and explain model strategies.Furthermore, it supports multiple ways to utilize these strategies for simplifying and improving the reference model.In collaboration with a large insurance company, we present a use case in automatic insurance acceptance, and show how professional data scientists were enabled to understand a complex model and improve the production model based on these insights.
Index Terms-Visual analytics, machine learning, explainable AI Ç

INTRODUCTION
W HILE modern machine learning (ML) techniques have great potential to solve a wide spectrum of real-world problems, some businesses have been reluctant to adopt this technology.Especially in high-risk environments, such as health care or the insurance sector, predictive performance alone is not sufficient.When critical decisions are made, we need to be able to hold ML models up to scrutiny.Either the model needs to be inherently interpretable, or the model has to be sufficiently explained using an external method.
This need is further exemplified by the surge of papers in which models are shown to be vulnerable to adversarial attacks.In these cases authors show that a small perturbation in the input (e.g., a single pixel in an image) can lead to unexpected, extreme changes in the output, often leading to absurd or incorrect predictions [1], [2].
In this paper, we show that complex models can be interpreted and analyzed through the identification and interpretation of different model strategies: different treatments by the model of distinct groups in the input data.As an example, fraud detection models typically classify data points into two categories: either fraudulent or not.However, there are likely many different ways to commit fraud, perhaps even ones that are unknown to experts.In this case, a single model needs to account for the many ways fraud can occur, leading it to become complex and uninterpretable.In this example, model strategies target the different ways of committing fraud, and can help to understand how the complex fraud model operates.
Finding these strategies is no trivial matter.Domain knowledge is required to ascertain the validity of these strategies.For this reason, we chose a visual analytics approach to actively involve data scientists in the analysis of their models and model strategies.
We present STRATEGYATLAS (see Fig. 1), a visual analytics system to enable understanding of complex models by identifying and interpreting different model strategies.These strategies give an intuitive global insight into the inner workings of the model.We apply dimensionality reduction to feature contribution vectors from wellknown explanation techniques (e.g., LIME [3], SHAP [4]).Points close to each other in the projection have similar feature contributions, which indicates a similar treatment by the model (i.e., a strategy).
These strategies can be used in various ways: to verify whether the model picks up on important concepts; to improve models if strategies do not match with the expectations and prior knowledge of data scientists; and to distill a simple and inherently interpretable surrogate model with minimal performance loss.Specifically, our main contributions are: StrategyMap, a projection based visualization approach to cluster data points based on similar treatment by the model (which correspond to model strategies); StrategyAtlas, a visual analytics approach to reveal what makes a cluster unique in a StrategyMap using contrastive explanation of the clusters, and assert their validity using domain knowledge; and a human-in-the-loop workflow to convert the assessed strategies into a set of corresponding interpretable models with comparable performance to the original reference model.
We collaborated with a leading insurance company in the Netherlands to obtain valuable insights into the relevance of explanations to data scientists, which guided our design decisions.We present a use case analyzing an operational machine learning model used for automatic acceptance of certain insurance policies.The use case shows that the analysts were enabled to understand a complex model and use that insight to improve their model used in production.Finally, we conclude with a reflection on our work and outline open research directions.

RELATED WORK
Explainable AI.There are two main approaches in the machine learning community to produce insights: either creating inherently interpretable models (e.g., GAM [5] and CORELS [6]), or explaining models post-hoc, using an external method (e.g., LIME [3] and SHAP [4]).Our work supports both approaches: model strategies are a post-hoc method for understanding models, and the system also enables creating interpretable models using strategy trees.
Visualization for Model Analysis.The increasing interest in explainable machine learning has led to an increasing demand for reliable visualization tools to support the understanding of ML models.It has become a prominent topic of research in the visualization community over the past decade [7].The majority of work focuses on explaining a single type of model (i.e., model-specific), such as Gamut [8], which investigates the role of interactive interfaces for model interpretation with additive models, and iForest [9] which enables the interpretation of predictions by Random Forest models.The models that received the most attention by far are neural networks [10], [11], [12], [13], [14], [15], [16], [17], [18].Strobelt et al. [10] for example tailor for sequence-tosequence models in the context of automatic translation, Garcia Caballero et al. [13] built a system specific to temporal LSTM networks for sleep staging, Ming et al. [14] targets Recurrent Neural Networks and reveal hidden memories in NLP tasks, and GANLab [16] promotes education and understanding of Generative Adversarial Networks.
The systems mentioned focus on providing explanations for one type of ML-model.Another approach is to use a model-agnostic method.We adapt this for STRATEGYATLAS, as it makes it more applicable in the real world where the models used come in all shapes and sizes.This approach is popular in Machine Learning research [3], [4], but adoption in the visualization community has been limited so far.Notable exceptions include Prospector [19], which uses 1D partial dependence as means to explore the prediction space, ExplainExplore [20] uses 2D partial dependence and incorporates feature contribution methods, and the What-if tool [21] enables testing hypotheses by means of data perturbations.These systems enable the understanding of single predictions (local perspective) whereas STRATEGYATLAS aims to build an understanding of the model as a whole (multiple instances at once, or global perspective).A visual explanation system that offers a global perspective is RuleMatrix [22], which induces and visualizes simplified derived decision rules.Their approach is to train a global surrogate, whereas our approach is to aggregate the result of local surrogates tailored for individual data instances, as recommended by Krause et al. [23].
Visualization Enabling Cluster Analysis.Cluster analysis is also a prevalent and related topic in visualization.These systems enable the discovery and understanding of clusters in the data with exploratory visual analysis.The Hierarchical Clustering Explorer [24] is an early example of such a system that relies heavily on interaction with dendrograms.It also features a heat map visualization to compare clusters, which is adopted in many more recent clustering systems [25], [26], [27].While this visualization offers a concise overview of the clusters, we instead opted for density plots as they can convey more information about feature values, and also enable us to separate clusters on the basis of other properties than just mean value (e.g., variance and multi-modality).In addition, we leverage unique properties of feature contribution vectors (e.g., generally only few features are important, and features share the same range that can be meaningfully sorted by value).
In recent works, clustering has also been applied to explain machine learning.Again, authors primarily target the explanation of neural networks.Rauber et al. [28] apply clustering on dimensionality reduced neuron activations in CNNs.They found that clusters often carry a semantic meaning, such as light digits on dark background versus dark digits on a light background.Zahavy et al. [29] apply the same technique to reinforcement learning agents playing video games.They found clusters corresponding to distinct playing strategies, like trapping the ball above the blocks in the game Breakout.Finally, DeepEyes [30] uses dimensionality reduced neuron activations for understanding the training process.Our goal is to highlight similar cluster structures, but preserve the model-agnostic nature to enable the analysis of a wide range of different classifiers.
A recent unpublished work that was released during our development also explores projections using feature contribution [31].However, the interpretation of clusters is limited to mean value bar charts only, and clusters are not related to patterns in the data, which is the main strength of our work.Another unpublished work called MELODY [32] also explores clusters in feature contribution vectors.However, these clusters are automatically constructed using a complex and opaque algorithm, which leaves no room for expert interpretation of the clusters.

PROBLEM DESCRIPTION
The primary goal of STRATEGYATLAS is to support data scientists in understanding the model they built.As we collaborate with a large insurance company, our primary focus will be data scientists in a business environment.The company asked us to support their data scientists in understanding their models, as they often had to choose between either interpretability (a small set of trusted models) or predictive performance (e.g., accuracy, F 1 score).

User Goals
We conducted six semi-structured interviews with data science teams interested in machine learning explanations at the insurance company.Guiding questions included whether and what they required explanation for, the type of data and model, and how explanations will benefit their daily work.We identified the following goals that our target users sought after when requiring explanations for their models: G1 Understand models to optimize performance (refine); G2 Enable experts to address problems and biases (diagnose); G3 Comply with regulations and customer requests (justify); G4 Reduce the time spent manually investigating classification results (decision-making).This is consistent with distinctions made in prior work [33].

User Tasks
Model strategies enable understanding a machine learning model by providing an intuitive representation that is not too simplistic (e.g., global feature importance), but not overly complex (e.g., showing all complex model internals, or feature contribution for each instance individually).By understanding how their model works, data scientists can make more informed decisions to achieve the goals mentioned previously: If strategies are deemed sound by domain experts, the important features in a strategy can be used to justify decisions to stakeholders (G3) and support decisions (G4).In addition, extra class labels can be added which could improve the generalization performance of the model (G1); If a strategy is not sound, this insight can be used to circumvent unexpected behavior (G2), or involved features can be removed to simplify the model (G1); Finally, strategies can be used to decompose a complex model into smaller, individually interpretable parts which makes it easier to make (G4) and justify (G3) decisions.To help experts to find and analyze model strategies to reach the user goals, we derived the following user tasks:  G3)? T5 Identify and remove features with minimal impact (G1).T6 Use strategies to decompose the complex model into simpler surrogate models (G1, G4).To support these tasks, we followed Brehmer and Munzner [34] to design the workflow shown in Fig. 2. It was verified by experts during the use case described in Section 7. A clear distinction is made between data (green) and model (blue) as early testing showed that experts tend to switch often between these perspectives and sometimes get confused.2. Workflow for STRATEGYATLAS.Arrows depict the typical flow of interaction, starting from the initial configuration of data and classifier.Uppercase words summarize the most important actions performed [34], and the corresponding user tasks are indicated.

Data
We specifically target tabular data only, and leverage some of its unique properties (e.g., user-interpretable feature names) to our advantage.Empirically, we found that such tabular datasets are much more commonly used in machine learning tasks (at our client) compared to images, text, and timeseries data.The system supports both numerical and categorical features, as well as a mix of the two.As we aim for model-agnosticity, any classification model can be used in STRATEGYATLAS, regardless of the number of classes.
Throughout this paper, we use the FICO HELOC Explainable Machine Learning Challenge dataset as an example [35].The goal is to predict whether an individual has been 90 days past due at least once in 24 months since opening a credit account (Bad) or not (Good).It contains 22 features (both numeric and ordinal) and 10,459 home equity credit applications.We trained an XGBoost classifier with 100 trees as the model we like to understand, as it achieved the highest F 1 score out of all the models we tested.

STRATEGYMAP APPROACH
Core to the STRATEGYATLAS system is the StrategyMap, which displays clusters corresponding to model strategies (Task T2) using a projection-based visualization approach.To achieve this, our system uses feature contribution techniques as a basis for model-agnostic machine learning understanding.These methods generate a vector of weights that indicate how much each feature has contributed to a single prediction.Any feature contribution techniques can be used, but we use LIME [3] as it has a straightforward interpretation (i.e., a feature contributes if a small change in feature value results in a large change in prediction), and relatively low computational cost.This low cost is beneficial, as we generate feature contribution vectors for every data point in the training dataset.
As the feature contribution values from LIME do not have a lower-or upper bound, all feature contribution vectors are max normalized.This can be thought of as "a feature x contributed 80% to this prediction", which is easier to explain to experts.This normalization step also improves the cluster separability by eliminating small differences in the prediction probability.For example, if two data points are predicted using the same subset of features but the prediction probability differs, the normalization step ensures they would still appear in the same cluster.
Next, the feature contribution vectors are projected down to two dimensions using UMAP [36].UMAP performs better at preserving some aspects of the global structure of the data and is generally faster than its competitors.It is also relatively stable which keeps the variation between consecutive runs small.However, StrategyMap is not limited to this choice: our initial prototypes used tSNE [37] and achieved similar results in terms of class separability.
Points close to each other in the projection will have similar feature contribution vectors, which indicates a similar treatment by the model (similar to [28], [29]).
As an example, consider the dataset of peppers in Fig. 3a.The clusters correspond to chili peppers (top) and bell peppers (bottom), of different colors.If a model predicts the ripeness of the pepper, indicated by the light and dark grey dot colors, only the feature Color need to be used.Hence only two clusters are noticeable in Fig. 3b, indicating that all items are classified using the same strategy.
However, if we instead would like to predict whether the vegetable can be represented as an emoji (e.g., or ) as shown in Fig. 4a, no two clusters of the same class can be classified using the same feature thresholds.All clusters have different feature contribution values, and hence all four clusters show up in the StrategyMap.
There is no guarantee that these model strategies are present in all datasets and model combinations.For instance, a StrategyMap for simple datasets such as the Iris or Titanic dataset will rarely show discernable clusters.The classes for these problems are sufficiently specified: no further structure can be inferred.Furthermore, simple models will rarely employ different model strategies.For instance, the feature contribution in linear models is the same for every data point in the dataset by design, and hence will show no clusters.However, in all these cases, the model is simple enough to be understood, and hence a global model explanation would not be needed.Our aim is to explain complex models.We argue that complexity in models will often be due to oversimplified, generic specification of classes.This is related to the Anna Karenina principle.Tolstoy's famous novel starts with "All happy families are alike; each unhappy family is unhappy in its own way."Similarly, for instance fraudulent behavior can have many different manifestations, and detecting these automatically leads to complex models with a variety of strategies.As an example of a truly complex model, we consider the FICO HELOC Explainable Machine Learning Challenge dataset and an XGBoost model with 100 decision trees (introduced in Section 3.3).It is the best model we managed to train and achieves an accuracy of 0.752, which is comparable to the accuracy achieved by the winners of the challenge (0.74) [38].A projection of this dataset and a StrategyMap projection of the XGBoost model are shown in Fig. 5.
Note that no cluster in the StrategyMap corresponds to a cluster in the DataMap.This occurs because all features in the data projection weigh equally, whereas low-contribution features will hardly affect the StrategyMap projection.
To conclude, the StrategyMap reveals structures in the model behavior that were previously challenging to detect.Clearly, certain data instances are treated differently from others by this model.However, it remains challenging to determine what constitutes these clusters.What features are used for classifications in a cluster?On what grounds are data instances treated differently?To answer these questions, and to verify whether it makes sense for the model to make these distinctions, we introduce STRATEGYATLAS.

STRATEGYATLAS
In this section, we describe how we translated the workflow (Fig. 2) into an interactive visual analytics system.Fig. 6 provides a high-level overview of our approach.The system offers three complementary methods to support interpreting model strategies: gradient heat maps (Fig. 6.2) enable the inspection of single features in the projection space; interactive density plots (Fig. 6.3) enable the analysis of clusters in terms of multiple features; and the cluster view (Fig. 6.4) helps to understand clusters by separating them from the other data.The main interface components are split into two rows: the top row shows components showing data aspects, whereas the bottom row pertains to model characteristics.For a demonstration of the system, we refer to the supplemental video.

Configuration View
The first step in the workflow is to set up the problem context to be analyzed.In this view (Fig. 6.1), any tabular dataset with numerical, categorical or mixed feature types can be added.Next, any classifier from the Python scikitlearn toolkit [39] or classifiers from other languages (e.g., R) and applications (e.g., KNIME, SAS Enterprise Miner) using the PMML format [40] are allowed.
If certain features are shown to have little relevance for classification, this view enables basic feature selection.A list shows all features annotated with the feature contribution density.A vertical bar plot of the mean value hides too many details (e.g., multimodality in the feature contribution values).However, displaying the distribution with a box plot or violin plot is more difficult to read due to the lack of correspondence between glyph area and feature contribution.We designed a bar plot using a Complement Cumulative Distribution Function (CCDF) that combines the strengths of both visualizations.An example is shown in Fig. 7, which shows NetFractionRevolvingBurden is almost always the most contributing feature.The contribution of NumInqLast6Mexcl7days is multimodal: for some data points it contributes around 60% and for others 80%.Either visualization can be chosen from the interface.

Projection Views
To identify clusters, the interface contains two maps: the DataMap, a UMAP projection of the dataset into two dimensions; and the StrategyMap, a UMAP projection of feature contribution vectors, in which clusters represent model strategies.We chose a scatterplot encoding to enable the expert data scientist to perform the clustering task.Other approaches (e.g., automated clustering) leaves no room for expert interpretation of the data, and this expert judgement is important, as the optimal granularity of clustering depends on the data and problem context.
The data points are colored using a greyscale colormap according to the predicted value by default.The colormap can also be applied based on ground truth values: a button underneath the legend enables quick toggling between the two (Task T1).This toggle can be used to estimate how many instances are incorrectly classified per cluster.The model performance may vary per cluster: one strategy cluster may be predicted almost perfectly whereas another may contain the majority of misclassifications.This helps to check the validity of a model strategy.We deliberately put this button front and center to reinforce the understanding that the system helps to understand the model predictions and not the ground truth.No conclusions should be drawn concerning the ground truth without considering the machine learning model makes mistakes too (correlations not causation).
To address overplotting, only a random subsample of 5,000 points is displayed.We found this to be representative for most datasets, and did not find additional model strategies beyond this limit.Additional data for each point is available via a customizable tooltip.

DataMap
To provide an overview of the entire dataset, the system includes a data projection view, shown in Fig. 6.2A.It also helps to relate whether strategy clusters in the StrategyMap are rooted in patterns in the data.For example, the left-most cluster in Fig. 6.2A is comprised of outliers with almost all feature values missing.The left-most cluster in Fig. 6.2B directly corresponds to those outliers.
The UMAP algorithm requires a distance function, and euclidean distance is not sufficient for dealing with mixed numerical and nominal data types.Instead, we opted for Gower distance [41]: a combination of Manhattan distance for numerical, and Dice distance for nominal features.The Gower distance dðp; qÞ between the vectors p and q is given by dðp; qÞ

<
: : (1) Note that for a dataset with only numerical features, the Gower distance is equivalent to the euclidean distance.

StrategyMap
This projection view includes a StrategyMap as introduced in Section 4, which enables Task T2.An example is shown in Fig. 6.2B.As feature contribution values are always numerical, regular euclidean distance can be used for the UMAP algorithm.

Gradient Heat Map Layer
This layer enables the inspection of the distribution of values of a specific feature in the data and feature contribution projection spaces (Task T3).This is achieved by interpolating the values of a chosen feature over the 2D projection space.Although interpolation only estimates the true value for each pixel in the projection space, it provides a sufficient heuristic for interpreting the space, that can be computed interactively (unlike more exact techniques [42]).
For numerical features, Inverse Distance Weighting (IDW) [43] is used for its applicability to irregularly spaced data.As UMAP projects points close together with similar values, it naturally enforces smooth transitions in the heat map, which is easier to navigate.
For nominal features, Voronoi tessellation is used by default, as no smooth transition exists between categories of a nominal feature.However, experts can freely choose between interpolation methods and adjust parameters such as the p parameter for IDW interactively, as shown in Fig. 8, by clicking the settings icon ( ).
We chose different colormaps to emphasize the difference between the data and model perspectives (Fig. 2).The data heat map uses a green sequential colormap, while the StrategyMap heat map uses a blue and red diverging colormap due to the divergent nature of feature contribution values (i.e., range from À1 to 1).The colormaps were chosen following the recommendations of prior work [44].
This technique can be applied to both the DataMap and StrategyMap.For example, in Fig. 6.2A the heat map shows  the left-most cluster has a low value for feature ExternalRis-kEstimate, while the rest of the data has similar values.For the StrategyMap, Fig. 8 shows the feature MSinceMostRecen-tInqexcl7days has a positive impact on the predictions of the two left-most clusters, a negative impact on the right-most cluster, and a varying impact in the remaining clusters.

Most-Contributing Feature Map
To get an overview of all gradient heat maps, we tried aggregating all heat maps and calculated the feature with the highest contribution for each pixel.This was implemented using a multi-pass GPU shader.We hypothesized the most contributing feature would differ per cluster, and hence this map would show a concise overview of the important features per cluster.However, we found that the most contributing feature often varies a lot, even within clusters, which makes the map difficult to interpret.An example is shown in Fig. 9.We decided not to include this feature in the final prototype.

Interaction
Selection enables experts to examine different subsets of data points.To this end, we chose lasso selection (e.g., draw a line around data points) to provide an intuitive interaction with full granular control.An example of lasso selection is shown in Fig. 1.2.The selection is linked and highlighted in all other views using the same blue color ( ) chosen to stand out amongst the other monochrome elements.
In addition to analyzing single clusters, the system enables contrastive explanation of what a cluster means: we can select two clusters and explore the differences between them (Task T4).With this, we follow the recommendations from social sciences [45] that adequate explanations of machine learning are ones that are contrastive with respect to another group.To this end, experts can make a secondary selection by using the right mouse button.The secondary selection is highlighted in a hot/cold contrasting red color ( ).

Data Density Plot Lists
To understand the cluster in terms of multiple features, the density plot lists enable comparison of the selection against the rest of the dataset.Two scrollable lists are shown for the data (Fig. 6.3A) and feature contribution values (Fig. 6.3B) respectively, echoing the separation of data and model from the projection views.
Ordinal and nominal feature densities are encoded using as a histogram by default (shown in Fig. 10.1), as it is a familiar encoding to our end user.The bars for the entire dataset are colored grey to blend into the background, while the primary and secondary selections are highlighted in the bottom fraction of each bar.
The density of numerical features is instead encoded with Kernel Density Estimation (KDE) plots by default (shown in Fig. 10.2), as the lines are less overwhelming in a list showing many feature densities at the same time.It uses an Epanechnikov kernel for its optimal efficiency and low computational cost [46].The density for the entire dataset is shown as a grey area chart, and the primary and secondary selection as lines to minimize occlusion for easy comparison (Task T3).The lines are colored bright blue and red to sharply contrast with the background.
Experts are enabled to switch between these visualizations, as some patterns in the data are easier to spot in one representation over the other.For example, sudden value spikes may be smoothed out in a KDE but visible in a histogram.In contrast, KDE can show more intricate details that would be hidden within a single histogram bar.
Other settings include the KDE kernel width parameter for the amount of smoothing, normalization of the selection densities, especially useful to enlarge and compare selections of few data points, and absolute values, which simplifies the interpretation of contribution values to important (high value) versus not important (low value).

Interaction
Experts are enabled to interact with the density plot lists in a variety of ways.The views can be sorted according to various properties, and each visualization supports selection by clicking or dragging.
Sorting.First, the expert can sort density plots by the mean value.This is especially useful for the contribution densities, to quickly spot what are the most contributing features globally, in the entire dataset.In addition, the plots can be sorted based on the mean of the primary selection and secondary selection to consider the most contributing features within those clusters (Task T3).
Second, the plots can be sorted by the standard deviation of the values.The results are sorted in ascending order, such that the feature with the lowest standard deviation is shown first.This is especially helpful when sorting data densities, and helps to figure out whether a selected cluster has (roughly) the same feature value for all data points.
Finally, the plots can be sorted by "selection separation".The two-sample Kolmogorov-Smirnov statistic is computed for each feature as where F 1;n and F 2;m are the empirical cumulative distribution functions of the first and the second selection, and sup the supremum function.This statistic indicates how likely samples from one selection are drawn from the same probability density function as another sample.In our case, it enables sorting on the difference between selections (Task T4).
Three variants are provided: comparing primary selection to all data, comparing the secondary selection to all data, and comparing the primary and secondary selection.Linking.By default, the data and contribution density plot lists can be navigated and sorted separately.However, to enable quick and easy comparison of the different densities (for instance, to analyze the feature values of the most contributing features), we provide a "link" button ( ) to synchronize the two lists.Once activated, every action in one list (i.e., scrolling, mouse-over and sorting) will also be applied to the other list.
Expanding.To make it easier to compare data and contribution density of every feature, an "expand" button ( ) enables to temporarily break the data and model separation in the interface and place the two lists side-by-side across the full height of the interface.Along with the link feature, it effectively turns the lists into a table.
Selection.All density plots support selection, which enables experts to select data points based on specific feature values or contribution values.The selection is highlighted in all other views in the system.For numerical features, brushing anywhere in the plot will select data points within the selected range.An example is shown in Fig. 10.For ordinal features, this range selection snaps to the categorical steps of the histogram.Finally, ordinal features can be selected by clicking on histogram bars.Any of these interactions can be performed with either the left or right mouse button, which controls whether the primary or secondary selection is used.

Cluster View
This view (shown in Fig. 6.4) enables experts to save selected clusters, and retrieve these at a later time.Additionally, decision trees can be trained to automatically compute the most relevant feature splits to explain the clusters, and to create surrogate models as an alternative to the complex reference model.
If a selection is made anywhere in the system, it can be stored using the add button ( ).Along with the selection, the most occurring predicted class within the cluster, and a user-defined label for the cluster are saved.The selected clusters are represented as tiles (Fig. 6.4) and include information such as the label and number of points in the cluster.When data points are selected that reside within the stored cluster, the tile is outlined with a thin blue or red border.If all points within the cluster are selected the tile is outlined with a thick border.Clicking on the tile will update the primary (left click) or secondary (right click) selection in the rest of the system.

Decision Tree Separation
The "train surrogate" button enables training a decision tree for each saved cluster, classifying data points as either within the strategy cluster or outside of it.We call these strategy trees.As our goal is to display a simple tree that is interpretable by the data scientist, we apply Minimal Cost-Complexity Pruning [47] with a ¼ 0:003.The goal of these strategy trees is twofold.
First, the strategy tree helps to understand strategy clusters.It shows the minimal subset of features required to discern the strategy cluster from others, as well as the order of importance.This provides a natural way of describing a strategy cluster (Task T3) and is a familiar encoding for our target user, the data scientist.
If a cluster tile is clicked, the strategy tree for that cluster is displayed, as shown in Fig. 11.The width of the links corresponds to the number of data points that end up in each child node, the color of the link is smoothly blended between gray and selection color based on the percentage of points in the child node that are selected.In this example, only four features are required to separate the cluster from the other data.The majority of points in this cluster have MSinceMostRecen-tInqexcl7days À3:5 (no months since last inquiry), External-RiskEstimate !74:5 (a low risk estimate, higher is better) and AverageMInFile !52:5 (are a long time customer).
In addition to displaying and understanding model strategies, our system includes ways to directly utilize the model strategies.After the data scientist has verified that the strategies are suitable for prediction and match their prior domain knowledge, the strategy trees can be used as a simple yet effective building block for a surrogate model that mimics the reference model.Such a surrogate model may be used as a more interpretable alternative to the complex model, striking a balance between complex black-box models and interpretable simple models.
Given that the model strategies are clusters of points that are classified similarly within the cluster, and differently from other clusters, we hypothesize that the behavior of this part of the model can be concisely represented with a shallow decision tree.For example, if only a few features are used to make predictions for a cluster, the corresponding strategy tree would only include those features.Prior work in the machine learning community has shown that tailoring smaller models to part of feature space can be effective and can even increase predictive accuracy [48].At the bottom of Fig. 6.4 a table is shown to enable the comparison of the reference model (Task T1) and the strategy trees.In addition, a third row shows the performance of a set of decision trees (the same number as strategy trees) trained on all data instead of per strategy cluster.This serves as a sanity check to ensure the strategy trees are indeed an improvement over normal decision trees.
The first column shows the F 1 score of these models on the test set.In the example, the strategy trees perform close to the reference model (1.1% difference) whereas the Random Forest performs a bit worse (2.8%), even though significantly fewer data are used to train the strategy trees.Next, the memory footprint of the model is shown as an estimate of the complexity of the model.The strategy trees are roughly 10% the size of the original model, while retaining most of its performance.Finally, the third column shows the percentage of overlap in predictions with the reference model.The strategy trees are more faithful to the complex model than the Random Forest.
These results are promising, but no definitive proof that the approach works.More research is needed to verify the effectiveness of this technique, which is outside the scope of this paper.

USE CASE 1: FICO CREDIT RISK
To illustrate the typical usage of STRATEGYATLAS, we explain some of the strategy clusters for the running example of the FICO HELOC dataset, shown in Fig. 6.2b.
There are two clusters for Good credit risk prediction.After selecting both clusters, the data density plots can be sorted based on selection separation to show in terms of which features the clusters differ.Fig. 6.3A shows that these clusters differ primarily in terms of feature MSinceMostRe-centInqexcl7days: the months since last inquiry.Negative values for this feature indicate missing data, so customers are treated differently based on either having no recent inquiry, or any positive number of recent inquiries.
To figure out how the model treats customers in these clusters differently, we sort the contribution density plots by selection separation.Fig. 6.3B shows that for customers with no recent inquiry ( ), the missing inquiry itself is the most important factor; all other features are less relevant.For customers that do have a recent inquiry ( ), the model is more vigilant, and uses (amongst others) ExternalRiskEstimate (higher is better), AverageMInFile (customer duration) and PercentTradesNeverDelq (number of delinquent trades).
The difference in relevance of the feature MSinceMostRe-centInqexcl7days is also supported by the heat map in Fig. 6.3B.Additionally, the most distinguishing features for the cluster are also present in the strategy tree for the cluster, which is displayed in Fig. 11.

USE CASE 2: AUTOMATIC INSURANCE ACCEPTANCE
To validate our approach in a real-world use case, we conducted a user study.The goal of this experiment was twofold: to test whether our system enables data scientists to understand the behavior of a complex machine learning model, and to test whether the experts were able to verify the validity of model strategy clusters using domain knowledge.We collaborated with data scientists from a large insurance company.The team created a predictive model for the purpose of aiding automatic acceptance of car insurances.The proprietary dataset contains 69 features (23 numeric, 13 ordinal and 33 nominal) and around 40,000 instances.They currently use a logistic regression model for the sake of transparency and interpretability (in spite of more complex models performing better).The model classifies data points into two categories: risk and non-risk.To explore whether complex models can be sufficiently explained using STRATE-GYATLAS, we trained a complex histogram-based gradient boosting model on their data for the data scientists to analyze.Compared to the logistic model, this model performs ten percent-points better, and is sufficiently different from the model developed by the team such that no confidential information can be inferred.

Participants & Procedure
Out of the four data scientists active on the project, three of them were prepared to participate in our study.The participants are between 30 and 42 years old, and are all full-time data scientists with at least five years experience in machine learning.They all primarily work with tabular data in their daily job, and only one of the participants reported having prior experience with XAI techniques.Finally, they had not used STRATEGYATLAS before the study.
Every session was conducted through a videoconferencing platform, and took two hours per participant.To start, each participant had signed a consent form and filled out a background questionnaire before the session.Next, we briefly introduced the system along with a demo using the FICO HELOC dataset, which took around 20 minutes.
We want to evaluate the system in a realistic scenario, and hence decided to structure the study as a field experiment as defined by Carpendale [49].By means of open ended questions we challenged the experts to explain aspects of their model, through which they would perform each of the user tasks.During this part, the data scientists interacted independently with the system, which ran on the infrastructure of the company to protect customer privacy.The thinkaloud method [49] was applied throughout the experiment, and all audio and screen activity were captured for further analysis.

Results
In the following subsections, we summarize our findings structured according to the user tasks introduced in Section 3.2.Figures are copied directly from the screen capture, but are redacted to protect sensitive information.We also name only a few key features, even though more features played a role during the analysis.

Task T1: Evaluating Model Performance
All experts started off with exploring the data itself before moving on to exploring the model.This helped them to get acquainted with the system.In this process, they toggled between prediction and ground truth coloring in the Strate-gyMap.Unexpectedly, two experts used this functionality to estimate the false positive rate among the non-risk points, and the false negative rate among the risk data points (i.e., the performance per cluster, shown in Fig. 12).
Even though not quite exact, this insight was sufficient for proceeding with caution, and it showed that they were aware the analyzed strategies were derived from an imperfect model, and thus did not reflect the true risk in the data.One expert said "Oh yeah hmm, it's not quite the truth, isn't it!"This insight is important, as overconfidence in explanations is a significant issue in ML [50], [51].
Later during the analysis, the experts created strategy trees for strategy clusters.To compare these strategy trees and the complex model they required a more exact metric, and here they did find and use the exact F 1 score at the bottom of the cluster view to evaluate the model performance and relative loss of the strategy trees.

Task T2: Verifying the Presence of Strategies
All experts immediately identified the same four clusters in the StrategyMap, and referred to these as distinct strategies.One expert said "I would say there are different strategies, because there are clusters that have different feature importance."The selections by expert two from the screen capture are shown in Fig. 13.The few outliers in between cluster two and three were not selected as they did not seem to belong to any of the clusters.
One expert was not very confident about cluster two, as the points were not as close together as the other clusters, and some sub-clustering was visible within the group.They rather considered it a 'remainder' group than a real model strategy.At a later point, the strategy tree corresponding to this cluster showed the expert it could be clearly separated from the rest of the data, and explained using a few features.This increased their trust in the strategy.
The experts were surprised to find that there was only one cluster for the risk prediction, and multiple for the nonrisk category.The label risk in the dataset was constructed manually using multiple indications, including proved fraud, prior defaulting and excessive number of previous claims.Hence, they expected the model to derive and use the same distinctions.However, they learned that due to data imbalance (e.g., the model has significantly more examples for the non-risk category compared to the risk category) the model learned the opposite correlation instead.To understand the model, the experts sorted the contribution density plots according to global mean value.This showed them the most important features for the entire model.Next, the experts used different approaches to interpret each of the clusters.We highlight the first three.The most used and preferred method was highlighting selections in the density plots along with the various sort options.
Cluster 1.All experts started analyzing the model by selecting the risk cluster first.There is only a single cluster for risk, which means that the model does not distinguish different types of risk, but rather different types of non-risk.
By sorting on the mean contribution value within the primary selection, the experts listed the most important features within this cluster (T3).One of these was the feature Car price.One expert linked the distribution plot lists and checked the data distribution for this feature.Surprisingly, the distribution of this feature was fairly similar across the risk and the non-risk data points.One expert said "Strange, if it is this important I also expect it to show something exceptional here."This occurs because the feature does not correlate with predicted risk on its own, but only in combination with other features; a typical trait of complex models.This became apparent when the expert analyzed the strategy tree for this cluster: Car price was shown at a deeper level in the decision tree, meaning the correlation was only present for a subset of the data (Fig. 15).Another important feature for this cluster was Claim-free years.For this feature the data distribution was different: data points in the risk cluster had much lower values compared to the rest of the data.Experts agreed this makes sense and matches with their expectations.
Cluster 2. This cluster is the largest of the non-risk clusters.The experts found that the feature Claim-free years was also important for this cluster.When consulting the data distribution plot, the values within the cluster were now much higher than average.One expert mentioned "this makes a lot of sense, as someone who has many claim-free years indeed poses much less risk." To check what made this cluster unique, one expert sorted the contribution density plots according to selection separation between cluster one and cluster two (T4).The clusters differed in terms of four features, amongst which Current car value.The contribution heat map for this feature (shown in Fig. 14) showed that this feature was much more important in this cluster than all others.One expert described this cluster the 'typical' non-risk cases: customers who have not submitted many claims in the past.For these customers, they found the model pays more attention to Current car value, compared to the other clusters (T3).
Cluster 3. The experts noticed that cases in cluster three have relatively low values for feature Claim-free years.Similar to the risk cluster, all customers in this cluster recently submitted a claim.However, if recent claims are a risk indicator, what makes this cluster non-risk?How does this nonrisk cluster differ from the risk cluster?
To find out, one expert trained a strategy tree for the cluster.This tree showed a very clear separation between data and clusters, with a fit of 94.5% and only 7 nodes.One of the important nodes in the tree selected cases based on high Customer duration.The expert continued to explore a couple of tree nodes, and verified the thresholds using selections in the data density plots.They concluded "with this tree you could identify this cluster in a pretty good way." To summarize (T3), this cluster could be labeled as reliable long term customers who did claim in the past.

Task T5 & T6: Utilizing Strategies
All experts trained strategy trees for each of the selected clusters during their analysis.
As they actively used strategy trees to understand model clusters, they gained familiarity with them, and were able to use them to explain predictions.This proves that strategy trees can be used as an inherently interpretable model.
The complex model, strategy trees, and reference Random Forest of participant 2 had an F 1 score of 65.5%, 63.1% and 54.7%.The estimated complexity, measured as the memory footprint in kilobytes, was 186kb, 9kb and 4kb respectively.The results of the other participants were similar.As the performance is so similar to the performance of the reference model, one expert said "in any case, I would rather use the [strategy] trees than the complex model".Another mentioned "the performance [of the strategy trees] is quite good", and seems to perform better than the production model, "while remaining interpretable, I think.".They were enthusiastic about using (aspects of the) strategy trees in their project.
As for feature selection, it was clear to the data scientists which features were globally relevant, only relevant to a select subset of the input data, and which were not important.Due to a technical limitation the feature selection dialog could not be used during the experiment, but experts were keen on experimenting with using different features in their production model, based on the insights from this study.

Reflection
In general, the participants positively received the system and reported STRATEGYATLAS helped them understand the complex model.The study sparked a lively discussion on the design decisions and possible improvements to their current model.They found the system easy to use, and learned to use the interface quickly.The experts were able to interpret and explain model strategy clusters and validate the behavior of the model by using domain knowledge.
We observed some interesting uses of the system.Even though data exploration is not an explicit goal of the system, all data scientists started with exploring the data before moving on to the model.Additions to support data exploration will benefit the workflow.
Next, density plot normalization was utilized more than expected.The densities for small selections are difficult to see, and this feature helps to compare them effectively.Based on this feedback, normalization could be enabled by default.
Experts had a personal preference for the absolute values feature in contribution density plots.One of the experts found the positive and negative correlations confusing and preferred this setting, whereas the other experts preferred seeing the unfiltered output from LIME.
Finally, none of the participants changed the interpolation method or corresponding settings during the experiment.We expect the default choices were sufficient for their analysis.

DISCUSSION & LIMITATIONS
The basic ideas presented in this paper are all simple in nature: 1) data and feature contribution projections using UMAP; 2) brushing and linking using lasso selections; 3) histogram and KDE density plots to analyze subsets of data and contribution; 4) persisting selection history to save clusters; and 5) training decision trees to discover differences between clusters.However, we have shown that combined they form a strong visual encoding that enables data scientists to identify and interpret model strategies, enabling them to understand the models they built.
There are, however, some limitations with regards to the applicability of our system.Our approach is specifically targeted at tabular data, as we found it to be much more prevalent in machine learning tasks (at our client).Hence, the application to other types of data falls outside the scope of this paper.Better visualization encodings can be used by  tailoring specifically to images [11].In addition, our approach requires user-interpretable feature names in the dataset.This excludes pre-processed datasets containing neural network latent representations.
STRATEGYATLAS is model-agnostic and supports a variety of models.However, it is currently limited to classification.For training strategy trees for each cluster, a clear distinction between classes is required.Next, smooth transitions in the output space (i.e., regression) inhibit the visual separation of clusters in the StrategyMap, which makes our approach less effective.In addition, we acknowledge that a solution tailored to a single type of classification model may yield deeper insights [28].However, our model-agnostic approach ensures it can be applied in practice, where the types of used models are numerous.
There is no inherent limitation on the number of features that can be represented.Both the DataMap and Strategy-Map are visually unaffected by the number of features, nor are the strategy trees in the cluster view.However, there is a practical limitation on the number of features shown in the density plot lists.In the use case, 69 features did not cause any problems, however we estimate an upper limit of about 100 features.This problem is mitigated in part by the sort feature, as well as integration with the browser search feature.Meanwhile, in all datasets we tested, we found that typically only the top 10-20 of features were relevant.
Due to overplotting concerns, the number of data points in both projection views is limited to 5,000.We argue this subsample of the dataset is sufficient for exposing strategy clusters in the model behavior.However, this subsampling step may not always faithfully represent all data.
Finally, the computation of all feature contribution vectors and UMAP projections takes around two minutes (on AMD Ryzen 5 3600X).Hence, it is currently not possible to interactively update parameters for the explanation and projection technique.The main bottleneck is computing the explanation technique; a more optimal inference of feature contribution vectors would benefit the system.

CONCLUSION
In this work, we presented STRATEGYATLAS: a visual analytics approach to enable a global understanding of complex machine learning models through the identification and interpretation of different model strategies.These model strategies are identified in our projection-based Strategy-Map visualization.Domain experts are enabled to ascertain the validity of these strategies: feature values and contributions can be analyzed using heat maps, density plots and decision tree abstractions.We explored the effectiveness of this approach using two use cases.First, we analyzed a model for a home equity line of credit dataset by FICO, and found distinct groups of customers that are treated differently by the model.Next, in collaboration with a large insurance firm, we applied the system in a real-world project for automatic insurance acceptance.The participants in the study received the system positively, and reported STRATE-GYATLAS helped them to understand the complex model.The study sparked a lively discussion on the initial choices and potential improvements to the production model.

Fig. 1 .
Fig. 1.Feature contribution vectors (1) are projected to show model strategies as clusters in the StrategyMap (2).Our proposed visual analytics system STRATEGYATLAS offers three methods to identify and interpret model strategies: (3A) a gradient heat map for individual features, (3B) interactive density plots for an overview of all features, and (3C) decision tree representations of strategies.

Fig.
Fig.2.Workflow for STRATEGYATLAS.Arrows depict the typical flow of interaction, starting from the initial configuration of data and classifier.Uppercase words summarize the most important actions performed[34], and the corresponding user tasks are indicated.

Fig. 3 .
Fig. 3. (a) Four clusters can be easily separated by ripeness (ripe/unripe) using the Color feature alone.(b) As far as the model is concerned, there are only two groups: green and red peppers.

Fig. 4 .
Fig. 4. (a) Four clusters can not be separated linearly by whether an emoji exists (emoji/non-emoji); different groups need different thresholds.(b) Hence, all four clusters show up in the StrategyMap.Note the snake-like structures are artifacts from oversimplified data.

Fig. 5 .
Fig. 5. (a) Data projection of the FICO dataset, (b) and corresponding StrategyMap projection showing five model strategies.

Fig. 8 .
Fig. 8. Interpolation and parameter choices for the contribution heat map.

Fig. 6 .Fig. 7 .
Fig. 6. (1) User interface of STRATEGYATLAS showing the main components: (1) the configuration view to select a dataset, model and feature contribution technique; (2) the projection views to highlight both (2A) clusters in the data, as well as (2B) model strategies as clusters in the feature contribution values; (3) a list of density plots to summarize and contrast selected clusters in terms of (3A) the data distribution, and (3B) the feature contribution values; and (4) the cluster view, to store selected clusters and train surrogate decision trees to explain and represent model strategies.A primary and secondary selection (blue and red respectively) highlight the properties of two clusters throughout the system.

Fig. 11 .
Fig. 11.Example of a strategy tree, highlighting the primary selection.

Fig. 12 .
Fig. 12. Toggling between prediction and ground truth point color enables experts to infer the model performance per cluster from the ratio of colors.

Fig. 13 .
Fig. 13.Selections in the StrategyMap made by the second expert.

Fig. 15 .
Fig. 15.Car price is only relevant in combination with other features.

Fig. 14 .
Fig. 14.The feature Current car value is most relevant in cluster 2.