Worker Filtering with Limited Supervision in Crowdsourcing Systems

In order to obtain high quality labels, it is important to recognize and tackle noisy workers in crowdsourcing applications. In particular, spam workers, who randomly assign labels to items, can greatly degrade the crowdsourced label quality. As such, we propose a semi-supervised worker filtering (SWF) approach to filter this type of workers among the crowd. The SWF model recognizes spam workers by utilizing a limited set of gold truths. An optimization based truth discovery framework, which minimizes the total errors reside workers' labels, is integrated with the semi-supervised worker filtering approach (SWF-TD) to infer the true labels for unlabeled items. The efficacy of the proposed methodology is demonstrated on both synthetic and real-world datasets. The experimental analysis on real world datasets showed that by using around 40% gold truths as priori knowledge, it is possible that SWF-TD approach provides similar performance to the fully labeled worker filtering model.