Cybersecurity Experimentation at Program Scale: Guidelines and Principles for Future Testbeds

Cybersecurity Experimentation is often viewed narrowly in terms of a single technology or experiment. This paper reviews the experimentation life-cycle for two large scale research efforts that span multiple technologies. We identify salient aspects of each cybersecurity program, and capture guidelines based on eight years of experience. Extrapolating, we identify four principles for building future experimental infrastructure: 1) Reduce the cognitive burden on experimenters when designing and operating experiments. 2) Allow experimenters to encode their goals and constraints. 3) Provide flexibility in experimental design. 4) Provide multifaceted guidance to help experimenters produce high-quality experiments. By following these principles, future cybersecurity testbeds can enable significantly higher-quality experiments.