Quantitative evaluation methodology for dynamic, Web-based collaboration tools

One of the primary hurdles to effectively evaluating adaptive software is that its very use alters the system from one moment to the next. Here, we developed and implemented an empirical-evaluation methodology that was successfully tested on an adaptive, Web-based collaboration tool. Subjects participated in an "open-book" exam, spending limited amounts of time on the Web researching a generic, non-technical topic, such as jazz or baseball, and were later tested on the information they had found. We employed objective, user-centered metrics to quantify subjects' ability to make enhanced Web searches when using the tool. Test results from 8 subjects showed that our approach could reliably measure differences in a user's performance for different collaboration algorithms. Our protocol demonstrated that history-dependent tools can be evaluated without large subject pools and without extensive database preparation if careful evaluation design is utilized.