Validation of Software Visualization Tools: A Systematic Mapping Study

Software visualization as a research field focuses on the visualization of the structure, behavior, and evolution of software. It studies techniques and methods for graphically representing these different aspects of software. Interest in software visualization has grown in recent years, producing rapid advances in the diversity of research and in the scope of proposed techniques, and aiding the application experts who use these techniques to advance their own research. Despite the importance of evaluating software visualization research, there is little work studying validation methods. As a consequence, it is usually difficult producing compelling evidence about the effectiveness of software visualization contributions. The goal of this paper is to study the validation techniques performed in the software visualization literature. We conducted a systematic mapping study of validation methods in software visualization. We consider 752 articles from multiple sources, published between 2000 and 2012, and study the validation techniques of the software visualization articles. The main outcome of this study is the lack in rigor when validating software visualization tool and techniques. Although software visualization has grown in interest in the last decade, it still lacks the necessary maturity to be properly and thoroughly evaluating its claims. Most article evaluations studied in this paper are qualitative case studies, including discussions about the benefits of the proposed visualizations. The results help understand the needs in software visualization validation techniques. They identify the type of evaluations that should be performed to address this deficiency. The specific analysis of SOFTVIS series articles shows that the specialized conference suffers from the same shortage.