Can recurrent models know more than we do?

Model interpretation is an active research area, aiming to unravel the black box of deep learning models. One common approach, saliency, leverages the gradients of the model to produce a per-input map highlighting the features most important for a correct prediction. However, saliency faces challenges in recurrent models due to the “vanishing saliency” problem: gradients decay significantly towards earlier time steps. We alleviate this problem and improve the quality of saliency maps by augmenting recurrent models with an attention mechanism. We validate our methodology on synthetic data and compare these results to previous work. This synthetic experiment quantitatively validate that our methodology effectively captures the underlying signal of the input data. To show that our work is valid in a real-world setting, we apply it to functional magnetic resonance imaging (fMRI) data consisting of individuals with and without a diagnosis of schizophrenia. fMRI is notoriously complicated and a perfect candidate to show that our method works even for complex, high-dimensional data. Specifically, we use our methodology to find the relevant temporal information of the subjects and connect our findings to current and past research.