Answer inference for crowdsourcing based scoring

Crowdsourcing is an effective paradigm in human centric computing for addressing problems by utilizing human computation power. While efforts have been made to study the crowdsourcing systems for labeling tasks such as classification, those for scoring tasks with continuous and correlative answers have not been well studied. In this paper, we propose two inference algorithms, MCE (Maximum Correlation Estimate) and WMCE (Weighted Maximum Correlation Estimate), to infer true answers based on answers submitted by workers. When estimating answers, WMCE algorithm assigns diverse weight to submitted answers of workers based on their quality while MCE algorithm assigns identical weight to submitted answers of all workers. For a fixed worker population, we reveal that the increase in task redundancy
¹
can improve accuracy of estimated answers but such improvement is limited within a certain level. We further show that WMCE algorithm can reduce the influence of this limitation better than MCE algorithm for the same crowdsourcing system. Simulation results validate our theoretical analysis and show that WMCE algorithm outperforms MCE algorithm in the accuracy of estimated answers.