Exclusion and Inclusion: A Model Agnostic Approach to Feature Importance in DNNs

Deep Neural Networks in NLP have enabled systems to learn complex non-linear relationships amongst words and phrases. Yet, one of the major bottlenecks for leveraging DNNs for real world applications is their characterization as black boxes. To solve this problem, we introduce a novel model agnostic algorithm which calculates phrase-wise importance of input features. We showcase generalizability of our method to a diverse set of tasks, by carrying out experiments for both Regression and Classification. We also observe that our approach works for short and long texts and is robust to outliers, implying that it captures the essential aspects of the input.