Studying Secure Coding in the Laboratory: Why, What, Where, How, and Who?

Software security is an area of growing concern, with over 192,000 known vulnerabilities in public software at the time of writing. Many aids to secure coding exist. Assessing the effectiveness of such aids in a laboratory environment is difficult. There are a number of concerns to address, such as recruitment issues and the level of instrumentation needed to perform an accurate measurement. Based on an extensive literature review of software development aids, we describe recent approaches to running laboratory studies, their characteristics, and their benefits and drawbacks. This paper should be of use to anyone planning to undertake coding studies with software developers.