The ergonomic criteria and the ISO/DIS 9241-10 dialogue principles: a pilot comparison in an evaluation task

This study compared the relative effectiveness of the Ergonomic Criteria and the ISO/DIS 9241-10 Dialogue Principles in an evaluation task. Three groups of participants (Criteria, n=6; ISO, n=5; Control, n=6) conducted an ergonomic inspection of a musical database application. Participants in the Control group relied solely on their individual judgement to conduct their evaluation. No significant correlation was found between the time spent conducting the evaluation and the number of problems uncovered (Kendall' Tau=0.324; Z corrected for ties= 1.853; p=0.0638). The median time spent evaluating the application was 52.5 (Mean rank=3.5), 89 (Mean rank=10.6), and 124 min (Mean rank=13.167), in the Control, ISO, and Criteria groups respectively (Kruskal–Wallis' H corrected for ties=11.762; p=0.0028). Multiple comparisons between groups revealed that only the Control and the Criteria groups differed significantly. The median number of usability problems uncovered by the participants in the Control, ISO and Criteria groups was 56.5 (Mean rank=6.25), 56 (Mean rank=7), and 85.5 (Mean rank=13.417) respectively (Kruskal–Wallis' H corrected for ties=7.18; p=0.0276). Again, multiple comparisons between groups revealed that only the Control and the Criteria groups differed significantly. The percentages of problems uncovered as a function of the number of evaluation reports cumulated showed similar trends for both the Control and the ISO groups. For the Criteria group, the percentage of problems uncovered with respect to the number of evaluation reports cumulated was higher. To summarise, participants using the Ergonomic Criteria spent significantly more time evaluating the application than the participants of the Control group. They also uncovered significantly more usability problems. However no significant differences appeared between the Control and the ISO groups as well as between the ISO and the Criteria groups.