An approach for the selection of evaluation methods for interactive adaptive systems using analytic hierarchy process

The evaluation of interactive adaptive systems is the source of many difficulties for novice evaluators as well as for expert ones. Issues arise in the need to identify the appropriate evaluation methods to be used for a given evaluation context. This paper proposes an approach for the choice of evaluation methods for interactive adaptive systems with regard to their suitability depending on specific evaluation contexts. A multicriteria decision making approach, namely the Analytic Hierarchy Process, is proposed for the selection methodology. The approach is based on a pair-wise comparison of a number of criteria that affect the choice of the appropriate evaluation methods. Four user-centred evaluation methods in conjunction with the layered evaluation for interactive adaptive systems are presented in the cases studies. These methods are: user-as-wizard, heuristic evaluation, focus group, and user test. The proposed approach is flexible in the sense that it can be used in the case of different evaluation methods. The results depend on the evaluation context where the methods will be used. Hence, we illustrate the proposed approach in the case of two evaluation contexts.