Thinking about thinking aloud: a comparison of two verbal protocols for usability testing

We report on an exploratory experimental comparison of two different thinking aloud approaches in a usability test that focused on navigation problems in a highly nonstandard Web site. One approach is a rigid application of Ericsson and Simon's (for original paper see Protocol Analysis: Verbal Reports as Data, MIT Press (1993)) procedure. The other is derived from Boren and Ramey's (for original paper see ibid., vol. 43, no. 3, p. 261-278 (2000)) proposal based on speech communication. The latter approach differs from the former in that the experimenter has more room for acknowledging (mm-hmm) contributions from subjects and has the possibility of asking for clarifications and offering encouragement. Comparing the verbal reports obtained with these two methods, we find that the process of thinking aloud while carrying out tasks is not affected by the type of approach that was used. The task performance does differ. More tasks were completed in the B and R condition, and subjects were less lost. Nevertheless, subjects' evaluations of the Web site quality did not differ, nor did the number of different navigation problems that were detected.