A Labeling Task Design for Supporting Recent Algorithmic Needs

Studies on supervised machine learning (ML) recommend involving workers from various backgrounds in training dataset labeling to reduce algorithmic bias. Moreover, sophisticated tasks for categorizing objects in images are necessary to improve ML performance, further complicating micro-tasks. This study aims to develop a task design incorporating the fair participation of people, regardless of their specific backgrounds or task’s difficulty. By collaborating with 75 labelers from diverse backgrounds for 3 months, we analyzed workers’ log-data and relevant narratives to identify the task’s hurdles and helpers. The findings revealed that workers’ decision-making tendencies were affected by the "community" that positively helps workers. Also, the machine’s feedback perceived by workers could make people easily engaged in works. Based on these findings, we suggest an extended human-in-the-loop approach that connects labelers, machines, and communities rather than isolating individual workers.