Reliability of Saliency Methods Used in Graph Neural Network Models

Graph neural networks (GNN) have recently been successfully used to combine data stored in electronic health records to identify individuals with similar phenotypes and model them to predict, for example, the discharge of patients from intensive care units or the best possible medical therapies based on similar patients' diagnoses. However, the explainability of the model is essential for a transparent and reliable application in medicine. This can be done using gradient-based methods such as saliency methods. While saliency methods in medicine are generally used in radiography to better understand what a model currently "observes" in regions of an image to find a prediction or make a classification, very little has been done in other areas also relevant in medicine such as GNNs applied in medical decision-making. To this end, we have tested gradient methods on GNNs and calculated their saliency scores with three different methods: i.) gradient, ii.) integrated gradients, and iii.) DeepLIFT. It is expected that the scores obtained with each method will be convergent. However, we found that less than 50% of the calculated scores converge. In addition, the quality of this convergence depends on whether the analysis is performed on the entire graph or on subgraphs. This result is relevant as it demonstrates that these methods are probably not fully reliable when applied to GNNs. Because of this, we propose an alternative strategy for using these scores: instead of using unique saliency scores, we suggest sampling the scores computed with different methods. In this way, it is not only possible to understand which input information influences the computation of the prediction, but it is also possible to compute a level of confidence of the predictions depending on the distribution of the scores, which makes it a human-centered method informing medical customers about the reliability of the model.