An experiment on subjective evolvability evaluation of object-oriented software: explaining factors and interrater agreement

Recent trends in software development have emphasized the importance of refactoring in preserving software evolvability. We performed two experiments on software evolvability evaluation, i.e. evaluating the existence of certain code problems called code smells and the refactoring decision. We studied the agreement of the evaluators. Interrater agreement was high for simple code smells and low for the refactoring decision. Furthermore, we analyzed evaluators' demographics and source code metrics as factors explaining the evaluations. The code metrics explained over 70% of the variation regarding the simple code smell evaluations, but only about 30% of the refactoring decision. Surprisingly, the demographics were not useful predictors neither for evaluating code smells nor the refactoring decision. The low agreement for the refactoring decisions may indicate difficulty in building tool support simulating real-life subjective refactoring decisions. However, code metrics tools should be effective in highlighting straightforward problems, e.g. simple code smells.