Modifications to the ISO 9186 Method for Testing Comprehension of Visualizations: Successes and Lessons Learned

Information visualizations are an increasingly popular approach to conveying information in a way that eases comprehension. However, rigorous methods for comprehension testing are needed for evaluation purposes. The International Organization for Standardization (ISO) 9186 method is the recognized standard for assessing comprehensibility of wayfinding and warning symbols but requires modification to maximize its utility for complex, health-related visualizations and low-literacy respondents. In this paper we present two case studies of research protocols deployed in health-related contexts that made modifications to the ISO 9186 method. Details of the modifications are provided. We describe successes and lessons learned as well as recommendations for future studies. Instructions that encouraged participants to give details as well as non-leading prompts for additional information helped elicit responses with sufficient granularity to make unambiguous judgments about participants’ level of comprehension.