Experiences in Scaling Field Studies of Software Developer Behavior

Most of our current understanding of how programmers perform various software maintenance and evolution tasks is based on controlled studies or interviews, which are inherently limited in size, scope, and realism. Replicating controlled studies in the field can both explore the findings of these studies in wider contexts and study new factors that have not been previously encountered in the laboratory setting. While replicating controlled studies in the field seems like an obvious next step in scientific progress, it is a step that has rarely been attempted, in part due to its complexity, which requires not only the industrial knowhow to implement a robust, scalable system, but the academic knowledge of how to design rigorous studies. In this talk, I will describe a few examples of successfully scaled studies, contrast them with less successful cases (including our own), and provide lessons learned. I will share the importance of collecting targeted information instead of generic logs, the insight that automated data collection paired with followup surveys is a powerful tool, and the nuances around what researchers can and cannot expect working developers to tolerate for the sake of research.