Assuring accuracy and impartiality in software design methodology comparison

Software product and process modelling can be used as bases for more accurate, objective, and repeatable comparison of software design methodologies (SDM's). In earlier work we demonstrated the use of a fixed feature comparison process (called CDM), a fixed (but extensible) framework of design methodology features (the Base Framework, BF), and a fixed process/product modelling formalism (based upon HFSP) in supporting such comparisons (X. Song, 1992; X. Song and L.J. Osterweil, 1994). We have obtained both broad, general SDM comparison results and sharp narrowly focussed SDM comparison results and have used them to confirm the power and validity of our approach. In the course of this work we have revised, modified, and extended the BF to improve its scope of applicability and its accuracy. We have also improved CDM by elaborating it down to lower levels of detail, and by being more specific about iteration conditions. In more recent work we have addressed the question of of how sensitive these comparisons and their results might be to the specific choice of process modelling formalism. We are learning that the comparison results remain surprisingly as the modelling formalism is changed.