Inside the black box: using Explainable AI to improve Evidence-Based Policies

The Evidence-Based Policy Movement (EBPM) advocates the intensive use of research and data to improve public policies and regulations. The last two decades saw the steady rise of the Big Data Era – a data-driven culture and mass production of information that could be an asset to the EBPM. However, the best models are complex black boxes and lack interpretability. Explainable Artificial Intelligence (XAI) is a booming field that brings explainability to those complex models and could fulfill Evidence-Based Policy needs. Shapley Additive Explanation (SHAP) proposes a unified framework of six previous XAI models and promises to bridge the gap between accuracy and interpretability. The Brazilian Electricity Regulatory Agency (ANEEL) aims at better policymaking by using consumer satisfaction as a regulatory tool. ANEEL uses IASC, a consumer satisfaction index, as a quality indicator that impacts companies’ price cap and profit margins. IASC intends to emulate competition in a monopolist market and ensure the consumer is a significant part of companies’ strategic objectives. To maximize the index’s potential and engage economic actors in the pursuit of consumer satisfaction drivers, ANEEL must increase the model’s predictive power and identify the most relevant features that will help achieve better results and cost-benefit. We propose a new model using Recurrent Neural Network and SHAP to accomplish this goal and keep transparency while reaching higher accuracy, producing actionable insights. We then compared this new model with the current model that uses structural equation modeling, analyzed caveats in employing SHAP, and assessed its benefits and advantages.