Systematic Labeling Bias: De-biasing Where Everyone is Wrong

Many real world classification problems use ground truth labels created by human annotators. However, observed data is never perfect, and even labels assigned by perfect annotators can be systematically biased due to poor quality of the data they are labeling. This bias is not created by the annotators from measurement error, but is intrinsic to the observational data. We present a method for de-biasing labels which simultaneously learns a classification model, estimates the intrinsic biases in the ground truth, and provides new de-biased labels. We test our algorithm on simulated and real data and show that it is superior to standard denoising algorithms, like instance weighted logistic regression.