Measuring success in non-trivial ways; how can we know that a DSS implementation has really worked

This paper details a 3-level analytic method for measuring a DSS implementation and for the early identification of problems with the deployment or the design, or subtle user misuses. Such problems may be small but nevertheless eventually fatal. The method incorporates domain-specific cognitive profiles of target users; measures of user navigation through both legacy systems and the target DSS; and ways of tracking the financial impact of the system at a level of detail normally not found among typical management reports. There are a number of unique features of each of these sub-methods. For example, the cognitive profiles measure the similarity of the user's spontaneous mental model of workflow to the underlying logic of the target DSS while at the same time measuring the lingering impact of legacy systems. Therefore, these profiles show movement (or the lack of) between paradigms during implementation. The user log analyses measure complexity of navigation through DSSs using system logs, and identify problems that would not be apparent by simply watching to see if users spend time at terminals. The financial impact measures provide an alternative to the site's usual measures while at the same time identifying lingering problem areas. This paper details each method in turn, and gives example data from actual sites. The paper concludes by showing how the results of each sub-method can be linked together, showing, for example, how paradigm shifts in users are affecting financial performance of shops.