Fairness Under Feature Exemptions: Counterfactual and Observational Measures

With the growing use of machine learning algorithms in highly consequential domains, the quantification and removal of disparity in decision making with respect to protected attributes, such as gender, race, etc., is becoming increasingly important. While quantifying disparity is essential, sometimes the needs of a business (e.g., hiring) may require the use of certain features that are critical in a way that any disparity that can be explained by them might need to be exempted. For instance, in hiring a software engineer for a safety-critical application, a coding-test score may be a critical feature that is weighed strongly in the decision even if it introduces disparity, whereas other features, such as name, zip code, or reference letters may be used to improve decision-making, but only to the extent that they do not add disparity. In this work, we propose a novel information-theoretic decomposition of the total disparity (a quantification inspired from counterfactual fairness) into two components: a non-exempt component which quantifies the part of the disparity that cannot be accounted for by the critical features, and an exempt component which quantifies the remaining disparity. This decomposition is important: it allows one to check if the disparity arose purely due to the critical features (inspired from the business necessity defense of disparate impact law) and also enables selective removal of the non-exempt component of disparity if desired. We arrive at this decomposition through canonical examples that lead to a set of desirable properties (axioms) that any measure of non-exempt disparity should satisfy. We then demonstrate that our proposed counterfactual measure of non-exempt disparity satisfies all of them. Our quantification bridges ideas of causality, Simpson's paradox, and a body of work from information theory called Partial Information Decomposition (PID). We also obtain an impossibility result showing that no observational measure of non-exempt disparity can satisfy all of the desired properties, which leads us to relax our goals and examine alternative observational measures that satisfy only some of these properties. We perform case studies to show how one can audit existing models as well as train new models while reducing non-exempt disparity.