I. Introduction
Continuous, automated surveillance systems incorporating machine learning models are becoming increasingly common in healthcare environments. These models can capture temporally dependent changes across multiple patient variables and enhance a clinician’s situational awareness by providing an early alarm of an impending adverse event. Among those adverse events, we are particularly interested in sepsis, which is a life-threatening medical condition contributing to one in five deaths globally [1] and stands as one of the most important cases for automated in-hospital surveillance. Recently, many machine learning methods have been developed to predict the onset of sepsis, utilizing electronic medical record (EMR) data [2]. A recent sepsis prediction competition [3] demonstrated the robust performance of XGBoost models [4], [5], [6]; meanwhile, Deep Neural Networks [7] are also commonly used. However, most approaches offer an alert adjudicator very little information pertaining to the reasons for the prediction, leading many to refer to them as “black box” models. Thus, model predictions related to disease identification, particularly for complex diseases, still need to be adjudicated (i.e., interpreted) by a clinician before further action (i.e., treatment) can be initiated. Among the aforementioned works, [6] provided one of the best attempts at identifying causality for their models’ predictions by reporting feature importance at a global level for all patients; still, this did not convey which features were most important in arriving at a given prediction for an individual patient. The common lack of interpretability of many clinical models, particularly those related to sepsis, suggests a strong need for principled methods to study the interactions among time series in medical settings.