By Topic

Combining Concurrent Think-Aloud Protocols and Eye-Tracking Observations: An Analysis of Verbalizations and Silences

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

3 Author(s)
Elling, S. ; Utrecht Inst. of Linguistics (UIL-OTS), Utrecht Univ., Utrecht, Netherlands ; Lentz, L. ; de Jong, M.

Research problem: Concurrent think-aloud (CTA) protocols are one of the dominant approaches of usability testing. However, there is still debate about the validity of the method, partly focusing on the usefulness and exhaustiveness of participants' verbalizations. The rise of eye-tracking technology sheds new light on this discussion, as participants' working processes can now be observed in more detail. Research questions: (1) What kinds of verbalizations do participants produce, and how do they relate to the information that can be directly observed using eye tracking? (2) What do eye movements reveal about cognitive processes at times when participants stop verbalizing? Literature review: Our study replicates an earlier study by Cooke (2010), who used a combination of CTA protocols and eye tracking in a small sample with experienced and highly educated participants to investigate the validity of CTA. Cooke's results suggest that the additional value of participants' verbalizations is limited: at least 77% of the verbalizations referred to things that could be easily observed with eye tracking. Methodology: We conducted a study in which 60 participants with different characteristics performed tasks on informational websites. During their task performance, they verbalized their thoughts, and simultaneously their eye movements were measured. The resulting think-aloud protocols were divided in verbalization units, which were coded into content types. Silences were registered, and eye movements during these silences were analyzed. Results and discussion: We found a different distribution of verbalization types than Cooke (2010) reported, with far more verbalizations where participants formulated doubts, judgments on the website, or expressions of frustration. In our study, verbalizations provided a substantial contribution in addition to the directly observable user problems. We measured a rather high percentage of silences (27%), during which participants most o- ten were scanning pages for information. During these silences, interesting observations could be made about users' processes and obstacles on the website. The implication of our study is that we now have a better understanding of the types of verbalizations that a CTA evaluation might generate. Further, we know that relevant usability observations can be made during silences. A limitation is that we do not know yet the influence of specific characteristics of the evaluation setting on the types of verbalizations and silences. Future research should focus on the influence of evaluation settings on the outcomes of an evaluation, in particular, the influence of characteristics of the participants who are involved in the study.

Published in:

Professional Communication, IEEE Transactions on  (Volume:55 ,  Issue: 3 )