Gaze-tracked crowdsourcing

When creating intelligent systems, we often need proper knowledge bases and resources annotated with metadata. Sometimes, we have no other option, than to utilize crowdsourcing, to acquire the data in necessary quantity. Crowdsourcing is a costly endeavor, always with space for improvements in task solving quantity and quality. Studies show that consideration of implicit feedback (behavior of workers during task solving) helps to improve the overall crowd output. Gaze-tracking is a powerful source of implicit feedback as it records user's activity outside typical feedback channels (e.g. clicking, scrolling, typing) and reveals a great deal of person's cognitive processes. This paper argues that gaze-tracking represents a potent feedback source even for crowdsourcing, as gaze-tracking technology becomes available for wider worker pools. The paper also presents an example case study demonstrating the use of the gaze-tracking during a typical crowdsourcing task - acquisition of training dataset for automated word sense disambiguation. Normally in such task, the worker explicitly selects a corresponding sense for a given word located in a text snippet and thus contributes to the dataset. With gaze-tracking involved, worker also shares us other information useful for dataset enrichment: worker's reading pattern (which may indicate confidence) and important sense-distinguishing words (e.g. contextual words that trigger worker's decisions).