Domain Generalization Study of Empirical Risk Minimization From Causal Perspectives

Empirical risk minimization (ERM) is a celebrated induction principle for developing data-driven models. However, ERM has received both pros and cons for its capability on domain generalization (DG). To this end, this paper attempts to study the success and failure of ERM at supervised DG classification tasks, both theoretically and empirically, with causal perspectives. In the theoretical aspect, we first explore different properties of a causal metric termed information flow, followed with discussing relationships between the information flow and the mutual information in the proposed causal graph. Next, we analyze the roles of the transformed causal feature and the transformed spurious feature on modeling performances. It reveals that the interaction between the spurious influencer and the transformed causal feature is the key determining the failure or success of ERM on DG. In the empirical study, we first simulate various DG settings based on the MNIST, Fashion MNIST, and CIFAR10 datasets. Next, we verify developed theories by testing three different neural network configurations in designed experiments. In addition, experiments based on real-world datasets are conducted to further consolidate key points of the proposed theories. To extend application benefits of the theoretical discoveries, a new risk minimization framework with a novel feature intervention for regulating ERM is proposed. It achieves DG improvements over ERM on real-world datasets of image segmentation, image classification, and text classification.