Evaluating Mitigation Approaches for Adversarial Attacks in Crowdwork

Crowdsourcing has emerged as a collaborative method to accomplish various tasks using open calls posted on platforms such as Amazon Mechanical Turk. Most requesters who post tasks on crowdsourcing platforms pay for each completed task, and the burden is on them to monitor quality. Most crowdworkers work for financial gain, so quickly completing tasks is paramount. As a result of crowdworker anonymity and a focus on this payment per completed task instead of payments based on accuracy, crowdsourcing has developed a reputation for questionable task quality. This paper discusses and evaluates four methods to detect malicious and adversarial crowdworkers. We conduct a series of empirical experiments that allow us to examine the effectiveness of these techniques in action. We find that each of the four suggested mitigation techniques has merit in improving quality and reducing adversarial attacks.