By Topic

Toward AI research methodology: three case studies in evaluation

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)
Cohen, P.R. ; Dept. of Comput. & Inf. Sci., Massachusetts Univ., Amherst, MA, USA ; Howe, A.E.

The roles of evaluation in empirical artificial intelligence (AI) research are described, in an idealized cyclic model and in the context of three case studies. The case studies illustrate the pitfalls in evaluation and the contributions of evaluation at all stages of the research cycle. Evaluation methods are contrasted with those of the behavioral sciences, and it is concluded that AI must define and refine its own methods. To this end, several experiment schemas and many specific evaluation criteria are described. Recommendations are offered in the hope of encouraging the development and practice of evaluation methods in AI. The first case study illustrates problems with evaluating knowledge-based systems, specifically a portfolio management expert system called FOLIO. The second study focuses on the relationship between evaluation and the evolution of the GRANT system, specifically, how the evaluations changed as GRANT's knowledge base was sealed up. Third, the cyclic nature of a given research model is examined

Published in:

Systems, Man and Cybernetics, IEEE Transactions on  (Volume:19 ,  Issue: 3 )