OnTac: Online task assignment for crowdsourcing

How to integrate labels from multiple labelers in order to obtain an accurate estimate of the ground truth is a major topic of crowdsourcing. One challenging issue is that, the labelers' abilities may vary significantly and the tasks distinguish each other in difficulties. Moreover, for a crowdsourcing system, task distributors have no idea in advance how many labels will be enough for each task. Consequently, an online task assignment mechanism based on the labeler expertise and question heterogeneousness becomes necessary. In this paper, we present such an online task assignment algorithm based on a probabilistic model consisting of both labeler abilities and question difficulties. We apply the online EM (Expectation Maximization) algorithm to make online estimations of system parameters, based on which we assign tasks adaptively. A series of simulation results have been demonstrated to show that our proposed scheme outperforms the conventional EM algorithm in efficiency and accuracy.