Bringing reputation-awareness into crowdsourcing

In crowdsourcing systems, worker selection is a challenging issue. Researchers have advocated the incorporation of reputation management into crowdsourcing systems to address this issue. However, current reputation-based decisions often result in concentrating the delegation of human intelligence tasks (HITs) to a small number of highly reputable workers to reduce risk. It conflicts with the main objective of crowdsourcing systems which is to promote mass collaboration. In this paper, we propose a situation-aware approach - SWORD - to enable existing reputation models to work in crowdsourcing systems. We holistically consider the objectives of all stake-holders in a crowdsourcing system (including requesters, workers, and system operators) to formulate the HIT allocation problem as a trade-off between quality and timeliness, and propose an efficient approach via constraint optimization to produce solutions to this problem with low computational complexity. Extensive simulations designed based on the actual conditions from Amazon's Mechanical Turk system demonstrates significant advantage of SWORD compared to existing approaches in improving social welfare.