Abstract:
The conventional wisdom in the field of statistical pattern recognition (SPR) is that the size of the finite test sample dominates the variance in the assessment of the p...Show MoreMetadata
Abstract:
The conventional wisdom in the field of statistical pattern recognition (SPR) is that the size of the finite test sample dominates the variance in the assessment of the performance of a classical or neural classifier. The present work shows that this result has only narrow applicability. In particular, when competing algorithms are compared, the finite training sample more commonly dominates this uncertainty. This general problem in SPR is analyzed using a formal structure recently developed for multivariate random-effects receiver operating characteristic (ROC) analysis. Monte Carlo trials within the general model are used to explore the detailed statistical structure of several representative problems in the subfield of computer-aided diagnosis in medicine. The scaling laws between variance of accuracy measures and number of training samples and number of test samples are investigated and found to be comparable to those discussed in the classic text of Fukunaga, but important interaction terms have been neglected by previous authors. Finally, the importance of the contribution of finite trainers to the uncertainties argues for some form of bootstrap analysis to sample that uncertainty. The leading contemporary candidate is an extension of the 0.632 bootstrap and associated error analysis, as opposed to the more commonly used cross-validation.
Published in: IEEE Transactions on Pattern Analysis and Machine Intelligence ( Volume: 25, Issue: 12, December 2003)
Center for Devices and Radiological Health, Food and Drug Administration, Rockville, MD, USA
Sergey V. Beiden received the MS degree in applied mathematics and physics from the Moscow Institute of Physics and Technology in 1990 and the PhD degree from the Russian Research Center Kurchatov Institute in 1993. He has held fellowships at the University of Sheffield (UK) and the University of West Virginia (US) doing first-principles calculations of complex processes in metals, alloys, and phase transitions. From 1999...Show More
Sergey V. Beiden received the MS degree in applied mathematics and physics from the Moscow Institute of Physics and Technology in 1990 and the PhD degree from the Russian Research Center Kurchatov Institute in 1993. He has held fellowships at the University of Sheffield (UK) and the University of West Virginia (US) doing first-principles calculations of complex processes in metals, alloys, and phase transitions. From 1999...View more
Department of Computer Science, George town University, Washington D.C., DC, USA
Marcus A. Maloof received the BS and MS degrees from the University of Georgia in 1989 and in 1992, respectively, and received the PhD degree from George Mason University in 1997. He was a postdoctoral fellow at the Institute for the Study of Learning and Expertise in Palo Alto, California, and a visiting scholar in the Center for the Study of Language and Information at Stanford University. In 1998, he joined the faculty...Show More
Marcus A. Maloof received the BS and MS degrees from the University of Georgia in 1989 and in 1992, respectively, and received the PhD degree from George Mason University in 1997. He was a postdoctoral fellow at the Institute for the Study of Learning and Expertise in Palo Alto, California, and a visiting scholar in the Center for the Study of Language and Information at Stanford University. In 1998, he joined the faculty...View more
Center for Devices and Radiological Health, Food and Drug Administration, Rockville, MD, USA
Robert F. Wagner received the BS in electrical engineering from Villanova University in 1959, the MA degree from Augustinian College in 1965, the MS degree in physics (1965) and the PhD degree in theoretical physics in 1969 from Catholic University of America. From 1970-1972, he did research in photo and electronuclear interactions at Ohio University. Since 1972, he has worked at the FDA's Center for Devices and Radiologi...Show More
Robert F. Wagner received the BS in electrical engineering from Villanova University in 1959, the MA degree from Augustinian College in 1965, the MS degree in physics (1965) and the PhD degree in theoretical physics in 1969 from Catholic University of America. From 1970-1972, he did research in photo and electronuclear interactions at Ohio University. Since 1972, he has worked at the FDA's Center for Devices and Radiologi...View more
Contents Center for Devices and Radiological Health, Food and Drug Administration, Rockville, MD, USA
Sergey V. Beiden received the MS degree in applied mathematics and physics from the Moscow Institute of Physics and Technology in 1990 and the PhD degree from the Russian Research Center Kurchatov Institute in 1993. He has held fellowships at the University of Sheffield (UK) and the University of West Virginia (US) doing first-principles calculations of complex processes in metals, alloys, and phase transitions. From 1999-2002, he worked at the US Food and Drug Administration on random-effects ROC analysis in breast and lung cancer screening and computer-aided diagnosis. He is currently working for the CARANA Corporation in Moscow on electricity market modeling and applications of machine learning and extreme value theory to forecasting.
Sergey V. Beiden received the MS degree in applied mathematics and physics from the Moscow Institute of Physics and Technology in 1990 and the PhD degree from the Russian Research Center Kurchatov Institute in 1993. He has held fellowships at the University of Sheffield (UK) and the University of West Virginia (US) doing first-principles calculations of complex processes in metals, alloys, and phase transitions. From 1999-2002, he worked at the US Food and Drug Administration on random-effects ROC analysis in breast and lung cancer screening and computer-aided diagnosis. He is currently working for the CARANA Corporation in Moscow on electricity market modeling and applications of machine learning and extreme value theory to forecasting.View more
Department of Computer Science, George town University, Washington D.C., DC, USA
Marcus A. Maloof received the BS and MS degrees from the University of Georgia in 1989 and in 1992, respectively, and received the PhD degree from George Mason University in 1997. He was a postdoctoral fellow at the Institute for the Study of Learning and Expertise in Palo Alto, California, and a visiting scholar in the Center for the Study of Language and Information at Stanford University. In 1998, he joined the faculty of Georgetown University, where he is an assistant professor in the Department of Computer Science. He is a member of AAAI, IEEE, and IEEE Computer Society. His research interests include online learning algorithms and their application to concept drift and computer security.
Marcus A. Maloof received the BS and MS degrees from the University of Georgia in 1989 and in 1992, respectively, and received the PhD degree from George Mason University in 1997. He was a postdoctoral fellow at the Institute for the Study of Learning and Expertise in Palo Alto, California, and a visiting scholar in the Center for the Study of Language and Information at Stanford University. In 1998, he joined the faculty of Georgetown University, where he is an assistant professor in the Department of Computer Science. He is a member of AAAI, IEEE, and IEEE Computer Society. His research interests include online learning algorithms and their application to concept drift and computer security.View more
Center for Devices and Radiological Health, Food and Drug Administration, Rockville, MD, USA
Robert F. Wagner received the BS in electrical engineering from Villanova University in 1959, the MA degree from Augustinian College in 1965, the MS degree in physics (1965) and the PhD degree in theoretical physics in 1969 from Catholic University of America. From 1970-1972, he did research in photo and electronuclear interactions at Ohio University. Since 1972, he has worked at the FDA's Center for Devices and Radiological Health on problems of statistical physics of the major medical imaging modalities, signal detection and statistical decision analysis, and computer-aided diagnosis, with more than 150 publications (shared Best Paper, 1983, IEEE Sonics and Ultrasonics). He is a fellow of the IEEE, SPIE, SPSE/IST (Imaging Societies), the Optical Society of America (OSA), and the American Institute of Medical and Biological Engineering (AIMBE). He was elected to the NIH/FDA Senior Biomedical Research Service in 1995.
Robert F. Wagner received the BS in electrical engineering from Villanova University in 1959, the MA degree from Augustinian College in 1965, the MS degree in physics (1965) and the PhD degree in theoretical physics in 1969 from Catholic University of America. From 1970-1972, he did research in photo and electronuclear interactions at Ohio University. Since 1972, he has worked at the FDA's Center for Devices and Radiological Health on problems of statistical physics of the major medical imaging modalities, signal detection and statistical decision analysis, and computer-aided diagnosis, with more than 150 publications (shared Best Paper, 1983, IEEE Sonics and Ultrasonics). He is a fellow of the IEEE, SPIE, SPSE/IST (Imaging Societies), the Optical Society of America (OSA), and the American Institute of Medical and Biological Engineering (AIMBE). He was elected to the NIH/FDA Senior Biomedical Research Service in 1995.View more
