System Evaluation Methodologies: Combined Multidimensional Scaling and Ordering Techniques

The results of a study designed to test a combination of methodologies for evaluation of systems, identification of a workable set of system evaluation attributes, and most importantly, specification of the interrelationships between these attributes as "seen" by the evaluators is reported. Twenty attributes were chosen which covered those areas deemed most important in systems evaluation and which reflected the thinking of a cross section of experts in the area of telecommunications. Multidimensional scaling and rank-order techniques were used to determine the interrelationships between attributes. The multidimensional scaling (MDS) results identified four dimensions: 1) system structure (from stable to dynamic), 2) system functions or service (service done on the system versus service performed by the system), 3) human-system interactions (human-system versus system-system interaction), and 4) system control (from "local" machine-oriented controls to "global" societal-oriented controls). Three different rank-ordering procedures were used, statistical, majority rule, and the Copeland method. This was followed by a partial ordering technique which synthesized these different viewpoints. The resultant partial order had ten levels with reliability deemed the most important attribute by all evaluators. This was followed at the next level by two attributes, performance and economy. The attribute deemed least important by all evaluators was observability. The MDS and rank-order results were then integrated into a unified conceptional framework. This framework reflected the importance of each attribute relative to its placement in the four-dimensional attribute space.