System Evaluation from Team Situation Awareness Perspective

Team situation awareness (TSA) is a crucial component for team performance, especially in complex and dynamic environment. Systems that support effective development of TSA are thus desired. However, methods of its evaluation are still limited. In this paper, we discuss the theoretical background of TSA and some issues regarding its measurement. We conducted an experiment to evaluate the viability of using a currently available TSA method, Questionnaires for Distributed Assessment of Team Mutual Awareness. Twenty male participants were recruited to perform simulated urban warfare missions in teams. The results show that one part of the method, taskwork mutual awareness measure, has acceptable predictive validity and sensitivity. However, the other two parts of the method, namely workload mutual awareness and teamwork mutual awareness, do not provide good gauges of TSA. This paper provides novel contributions to the area of TSA, and also useful information for practitioners to select a TSA measurement tool as a part of their system evaluation methods.