Automation bias and system design: a case study in a medical application

We discuss the results of an interdisciplinary study of computer-aided decision making in cancer screening. We found evidence of "automation bias" - roughly, over-reliance on automation, which may degrade users' performance. While such effects are well known, they have not been previously reported for this application with expert clinicians, and are not generally considered in the literature. One lesson from our study was that these potentially important effects easily go unnoticed with common assessment methods. Implications for designers of computer aided decision making include that: a) it may be necessary to calibrate tool design for a range of different levels of user skills; b) an "expert system" approach to computerised aid design - building a "better replica" of a human expert - may be counterproductive by making the aid weak in those very areas where humans need help; c) HCI design risks focusing on usability of the physical human-computer interface, but the critical issues in design concern the cognitive effects (e.g. changes of decision thresholds) a computer tool may have on users; d) users' subjective assessments of a computer aid may be misleading: people may judge a tool helpful when their decision-making performance is actually being hampered.