Measuring Program Comprehension: A Large-Scale Field Study with Professionals

This paper is published in IEEE Transaction on Software Engineering (DOI: 10.1109/TSE.2017.2734091). Comparing with previous programming comprehension studies that are usually in controlled settings or have a small number of participants, we perform a more realistic investigation of program comprehension activities. To do this, we extend our ActivitySpace framework to collect and analyze Human-Computer Interaction (HCI) data across many applications (not just the IDEs). We collect 3,148 working hour data from 78 professional developers in a field study. We follow Minelli et al.'s approach to assign developers' activities into four categories: navigation, editing, comprehension, and other. Then we measure comprehension time by calculating the time that developers spend on program comprehension. We find that on average developers spend ~58% of their time on program comprehension activities, and that they frequently use web browsers and document editors to perform program comprehension activities. We also investigate the impact of programming language, developers' experience, and project phase on the time that is spent on program comprehension.