A Generalized Worker-Task Specialization Model for Crowdsourcing: Optimal Limits and Algorithm

Crowdsourcing has emerged as an effective platform to label data with low cost by using non-expert workers. However, inferring correct labels from multiple noisy answers on data has been a challenging problem, since the quality of answers varies widely across tasks and workers. Many existing prior works have assumed a simple model where the order of workers in terms of their reliabilities is fixed across tasks, and focused on estimating the worker reliabilities to aggregate responses with different weights. We propose a highly general crowdsourcing model in which the reliability of each worker can vary depending on the type of a given task, where the number of types d can scale in the number of tasks. In this model, we characterize the optimal sample complexity to correctly infer the unknown labels within any given accuracy, and propose an algorithm achieving the order-wise optimal result. We conduct experiments on synthetic and real datasets, and show that our algorithm outperforms the existing ones developed based on strict model assumptions.
¹