User and R&D specialist evaluation of decision-support systems

The research reported represents an initial step toward developing an empirical knowledge base regarding how user and specialist group evaluate the utility of decision-support system (DSS) prototypes. To obtain such information a multiattribute evaluation hierarchy was developed that contained the kinds of factors that theoretically should (and empirically did) discriminate between the utility judgments of the participating user and specialist representatives. This was used to evaluate different prototypes and thereby collect data regarding what factors different user and specialist groups generally consider most/least important when making utility judgments. A standardized questionnaire recently used by substantive experts (i.e. potential users) and technical representatives (i.e. R&D specialists) to evaluate five DSS prototypes for US Air Force tactical decision-making is described. Reliability and validity measures obtained after analyzing the evaluation participants' responses indicate that the questionnaire was an acceptable instrument for measuring subjective assessments of DSS prototypes. In addition, there was support for the theoretical position that people use themselves (i.e. their knowledge, skills, needs, etc.) as a reference point when evaluating the potential utility of DSS prototypes.