Measuring IS: toward a US standard

At the urging of intelligent systems consumers, including industry groups and the US military, the US National Institute of Standards and Technology (NIST) is taking the first steps toward creating standard measures for intelligent systems (IS). The need for intelligent systems in the public and private sectors is complicated by many unanswered questions. Among those questions are how to precisely define intelligent systems and how to measure and compare the capabilities that particular systems claim to provide. NIST's initial approach to establishing performance metrics is twofold and attempts to address both pragmatic and theoretical concerns. At the end of July, NIST unveiled its test course for mobile robots at AAAI in Austin. Two weeks later, it held its first performance metrics workshop, where the idea of such a testbed was just one among several controversial issues. Both the workshop and the test-course competition inspired debate over the value of focusing on what's practical versus what's possible when it comes to system intelligence.