Process Design to Use Amazon MTurk for Cognitively Complex Tasks

Amazon Mechanical Turk (MTurk) is a very popular method for researchers to complete large volumes of repeatable, cognitively simple tasks such as data labeling. However, MTurk is currently challenging to use for more cognitively complex tasks. This article presents a procedure to help researchers interested in using MTurk to complete large volumes of repeatable, cognitively complex tasks. By combining batches, dummy variables, and worker qualification, it is possible for researchers to design a simple one-click user interface to assign the same task to a targeted set of MTurk workers and gather data from all these workers in one place. By scanning the marketplace for good workers and assigning tasks only to these workers, it is possible for researchers to greatly simplify the use of MTurk for cognitively complex tasks.