Evaluating emerging software development technologies: lessons learned from assessing aspect-oriented programming

Determining whether a new software development technique is useful and usable is a challenging taste. Various flavors of empirical study may be used to help with this task, including surveys, case studies, and experiments. Little guidance is available within the software engineering community to help choose among these alternatives when assessing a new and evolving software development technique within some cost bounds. We faced this challenge when assessing a new programming technique called aspect-oriented programming. To assess the technique, we chose to apply both a case study approach and a series of four experiments because we wanted to understand and characterize the kinds of information that each approach might provide. We describe and critique the evaluation methods we employed, and discuss the lessons we have learned. These lessons are applicable to other researchers attempting to assess new programming techniques that are in an early stage of development.