Abstract:
This article proposes a framework for evaluating and modeling perception systems, motivated by the need to develop testing scenarios for verification and validation of au...Show MoreMetadata
Abstract:
This article proposes a framework for evaluating and modeling perception systems, motivated by the need to develop testing scenarios for verification and validation of autonomous driving systems operating in various driving environment perception approaches, including both ego-vehicle centric perception and cooperative perception with enabled connectivity. The proposed perception system evaluation and modeling approach is probabilistic, with perception failures and errors encoded as stochastic processes and accounts for the operation domain. The perception error model is parameterized to consider both spatial and temporal aspects in the offline evaluation process. The proposed method exhibits well-fitting performance on the model of the perception error pattern based on evaluation results in various virtual and real traffic data with several benchmark perception algorithms.
Published in: IEEE Transactions on Intelligent Vehicles ( Volume: 9, Issue: 1, January 2024)
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Chen Sun received the B.Eng. degree in automation from the University of Electronic Science and Technology of China, Chengdu, China, in 2014, the M.A.Sc degree in electrical and computer engineering from the University of Toronto, Toronto, ON, Canada in 2017, and the Ph.D. degree in mechanical and mechatronics engineering from University of Waterloo, Waterloo, ON, Canada, in 2022. He is currently a Postdoctoral Fellow wit...Show More
Chen Sun received the B.Eng. degree in automation from the University of Electronic Science and Technology of China, Chengdu, China, in 2014, the M.A.Sc degree in electrical and computer engineering from the University of Toronto, Toronto, ON, Canada in 2017, and the Ph.D. degree in mechanical and mechatronics engineering from University of Waterloo, Waterloo, ON, Canada, in 2022. He is currently a Postdoctoral Fellow wit...View more
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Yaodong Cui (Graduate Student Member, IEEE) received the B.S. degree in automation from Chang'an University, Xi'an, China, in 2017, and the M.Sc. degree in systems, control and signal processing from the University of Southampton, Southampton, U.K., in 2019. He is currently working toward the Ph.D. degree with the Waterloo Mechatronic Vehicle Systems (MVS) Lab, Department of Mechanical Engineering, University of Waterloo,...Show More
Yaodong Cui (Graduate Student Member, IEEE) received the B.S. degree in automation from Chang'an University, Xi'an, China, in 2017, and the M.Sc. degree in systems, control and signal processing from the University of Southampton, Southampton, U.K., in 2019. He is currently working toward the Ph.D. degree with the Waterloo Mechatronic Vehicle Systems (MVS) Lab, Department of Mechanical Engineering, University of Waterloo,...View more
Huawei Technologies Canada Company Ltd., Ottawa, ON, Canada
Ngoc-Dũng Dào received the Ph.D. degree from the University of Alberta, Edmonton, AB, Canada, in 2007. From 2007 to 2010, he was with Toshiba Research Europe, Bristol, U.K., Since 2010, he has been with Huawei Technologies Canada. He is currently a Principle Engineer with Huawei Technologies Canada. His research interests include 5G and beyond cellular network architectures, data analytics, vehicle communications, IoT, an...Show More
Ngoc-Dũng Dào received the Ph.D. degree from the University of Alberta, Edmonton, AB, Canada, in 2007. From 2007 to 2010, he was with Toshiba Research Europe, Bristol, U.K., Since 2010, he has been with Huawei Technologies Canada. He is currently a Principle Engineer with Huawei Technologies Canada. His research interests include 5G and beyond cellular network architectures, data analytics, vehicle communications, IoT, an...View more
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Reza Valiollahi Mehrizi received the Master's degree and the Ph.D. degree from the University of Waterloo, Waterloo, ON, Canada, in 2017 and 2021, respectively. He is currently a statistician. He is also a Data Scientist and Statistical Researcher with the Mechatronic Vehicular System Lab, University of Waterloo. His expertise lies in deep learning object detection, semantic image segmentation, computer vision, and natura...Show More
Reza Valiollahi Mehrizi received the Master's degree and the Ph.D. degree from the University of Waterloo, Waterloo, ON, Canada, in 2017 and 2021, respectively. He is currently a statistician. He is also a Data Scientist and Statistical Researcher with the Mechatronic Vehicular System Lab, University of Waterloo. His expertise lies in deep learning object detection, semantic image segmentation, computer vision, and natura...View more
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Mohammad Pirani (Senior Member, IEEE) received the M.A.Sc degree in electrical and computer engineering and the Ph.D. degree in mechanical and mechatronics engineering, the University of Waterloo, Waterloo, ON, Canada, in 2014 and 2017, respectively. He is currently a Research Assistant Professor with the Department of Mechanical and Mechatronics Engineering, University of Waterloo. He held postdoctoral Researcher positio...Show More
Mohammad Pirani (Senior Member, IEEE) received the M.A.Sc degree in electrical and computer engineering and the Ph.D. degree in mechanical and mechatronics engineering, the University of Waterloo, Waterloo, ON, Canada, in 2014 and 2017, respectively. He is currently a Research Assistant Professor with the Department of Mechanical and Mechatronics Engineering, University of Waterloo. He held postdoctoral Researcher positio...View more
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Amir Khajepour (Senior Member, IEEE) received the Ph.D. degree from the University of Waterloo, Waterloo, ON, Canada, in 1996. He is currently a Professor of mechanical and mechatronics engineering and the Canada Research Chair of Mechatronic Vehicle Systems with the University of Waterloo, where he is also the Waterloo Center for Automotive Research Executive Director. He is an Expert in systems modeling and control of d...Show More
Amir Khajepour (Senior Member, IEEE) received the Ph.D. degree from the University of Waterloo, Waterloo, ON, Canada, in 1996. He is currently a Professor of mechanical and mechatronics engineering and the Canada Research Chair of Mechatronic Vehicle Systems with the University of Waterloo, where he is also the Waterloo Center for Automotive Research Executive Director. He is an Expert in systems modeling and control of d...View more
Contents MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Chen Sun received the B.Eng. degree in automation from the University of Electronic Science and Technology of China, Chengdu, China, in 2014, the M.A.Sc degree in electrical and computer engineering from the University of Toronto, Toronto, ON, Canada in 2017, and the Ph.D. degree in mechanical and mechatronics engineering from University of Waterloo, Waterloo, ON, Canada, in 2022. He is currently a Postdoctoral Fellow with the Mechatronic Vehicle Systems Laboratory (MVSL), University of Waterloo. His research interests include end-to-end autonomous driving, safety validation for cyber-physical systems, planning and control for robots.
Chen Sun received the B.Eng. degree in automation from the University of Electronic Science and Technology of China, Chengdu, China, in 2014, the M.A.Sc degree in electrical and computer engineering from the University of Toronto, Toronto, ON, Canada in 2017, and the Ph.D. degree in mechanical and mechatronics engineering from University of Waterloo, Waterloo, ON, Canada, in 2022. He is currently a Postdoctoral Fellow with the Mechatronic Vehicle Systems Laboratory (MVSL), University of Waterloo. His research interests include end-to-end autonomous driving, safety validation for cyber-physical systems, planning and control for robots.View more
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Yaodong Cui (Graduate Student Member, IEEE) received the B.S. degree in automation from Chang'an University, Xi'an, China, in 2017, and the M.Sc. degree in systems, control and signal processing from the University of Southampton, Southampton, U.K., in 2019. He is currently working toward the Ph.D. degree with the Waterloo Mechatronic Vehicle Systems (MVS) Lab, Department of Mechanical Engineering, University of Waterloo, Waterloo, ON, Canada. His research interests include sensor fusion, perception for the intelligent vehicle, and driver emotion detection.
Yaodong Cui (Graduate Student Member, IEEE) received the B.S. degree in automation from Chang'an University, Xi'an, China, in 2017, and the M.Sc. degree in systems, control and signal processing from the University of Southampton, Southampton, U.K., in 2019. He is currently working toward the Ph.D. degree with the Waterloo Mechatronic Vehicle Systems (MVS) Lab, Department of Mechanical Engineering, University of Waterloo, Waterloo, ON, Canada. His research interests include sensor fusion, perception for the intelligent vehicle, and driver emotion detection.View more
Huawei Technologies Canada Company Ltd., Ottawa, ON, Canada
Ngoc-Dũng Dào received the Ph.D. degree from the University of Alberta, Edmonton, AB, Canada, in 2007. From 2007 to 2010, he was with Toshiba Research Europe, Bristol, U.K., Since 2010, he has been with Huawei Technologies Canada. He is currently a Principle Engineer with Huawei Technologies Canada. His research interests include 5G and beyond cellular network architectures, data analytics, vehicle communications, IoT, and multimedia. He has served on the Editorial boards of IEEE Communications Survey and Tutorials, IEEE Transactions on Vehicular Technology, and IEEE Communications Letters. He is currently the Series Editor (IoT and Sensor Networks) of IEEE Communications Magazine.
Ngoc-Dũng Dào received the Ph.D. degree from the University of Alberta, Edmonton, AB, Canada, in 2007. From 2007 to 2010, he was with Toshiba Research Europe, Bristol, U.K., Since 2010, he has been with Huawei Technologies Canada. He is currently a Principle Engineer with Huawei Technologies Canada. His research interests include 5G and beyond cellular network architectures, data analytics, vehicle communications, IoT, and multimedia. He has served on the Editorial boards of IEEE Communications Survey and Tutorials, IEEE Transactions on Vehicular Technology, and IEEE Communications Letters. He is currently the Series Editor (IoT and Sensor Networks) of IEEE Communications Magazine.View more
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Reza Valiollahi Mehrizi received the Master's degree and the Ph.D. degree from the University of Waterloo, Waterloo, ON, Canada, in 2017 and 2021, respectively. He is currently a statistician. He is also a Data Scientist and Statistical Researcher with the Mechatronic Vehicular System Lab, University of Waterloo. His expertise lies in deep learning object detection, semantic image segmentation, computer vision, and natural language processing as well as experimental designs and survival analysis. Throughout his academic and professional career, he has demonstrated a passion for using statistical methods to solve complex data problems and advancing the field of statistics and AI.
Reza Valiollahi Mehrizi received the Master's degree and the Ph.D. degree from the University of Waterloo, Waterloo, ON, Canada, in 2017 and 2021, respectively. He is currently a statistician. He is also a Data Scientist and Statistical Researcher with the Mechatronic Vehicular System Lab, University of Waterloo. His expertise lies in deep learning object detection, semantic image segmentation, computer vision, and natural language processing as well as experimental designs and survival analysis. Throughout his academic and professional career, he has demonstrated a passion for using statistical methods to solve complex data problems and advancing the field of statistics and AI.View more
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Mohammad Pirani (Senior Member, IEEE) received the M.A.Sc degree in electrical and computer engineering and the Ph.D. degree in mechanical and mechatronics engineering, the University of Waterloo, Waterloo, ON, Canada, in 2014 and 2017, respectively. He is currently a Research Assistant Professor with the Department of Mechanical and Mechatronics Engineering, University of Waterloo. He held postdoctoral Researcher positions at the University of Toronto during 2019–2021 and KTH Royal Institute of Technology, Sweden, during 2018–2019. His research interests include resilient and fault-tolerant control, networked control systems, and multi-agent systems.
Mohammad Pirani (Senior Member, IEEE) received the M.A.Sc degree in electrical and computer engineering and the Ph.D. degree in mechanical and mechatronics engineering, the University of Waterloo, Waterloo, ON, Canada, in 2014 and 2017, respectively. He is currently a Research Assistant Professor with the Department of Mechanical and Mechatronics Engineering, University of Waterloo. He held postdoctoral Researcher positions at the University of Toronto during 2019–2021 and KTH Royal Institute of Technology, Sweden, during 2018–2019. His research interests include resilient and fault-tolerant control, networked control systems, and multi-agent systems.View more
MVS Lab, Department of Mechanical and Mechatronics Engineering, University of Waterloo, Waterloo, ON, Canada
Amir Khajepour (Senior Member, IEEE) received the Ph.D. degree from the University of Waterloo, Waterloo, ON, Canada, in 1996. He is currently a Professor of mechanical and mechatronics engineering and the Canada Research Chair of Mechatronic Vehicle Systems with the University of Waterloo, where he is also the Waterloo Center for Automotive Research Executive Director. He is an Expert in systems modeling and control of dynamic systems, and he has developed an extensive research program that applies his expertise to several key multidisciplinary areas. His research has resulted in several patents, technology transfers, and two startup companies. He has authored or coauthored more than 350 journal and conference publications, as well as five books and seven book chapters. Dr. Khajepour is a Fellow of the Engineering Institute of Canada and the American and Canadian Society of Mechanical Engineering. He was the recipient of the Engineering Medal from Professional Engineers Ontario. He is an Associate Editor for the International Journal of Vehicle Autonomous Systems and International Journal of Powertrains.
Amir Khajepour (Senior Member, IEEE) received the Ph.D. degree from the University of Waterloo, Waterloo, ON, Canada, in 1996. He is currently a Professor of mechanical and mechatronics engineering and the Canada Research Chair of Mechatronic Vehicle Systems with the University of Waterloo, where he is also the Waterloo Center for Automotive Research Executive Director. He is an Expert in systems modeling and control of dynamic systems, and he has developed an extensive research program that applies his expertise to several key multidisciplinary areas. His research has resulted in several patents, technology transfers, and two startup companies. He has authored or coauthored more than 350 journal and conference publications, as well as five books and seven book chapters. Dr. Khajepour is a Fellow of the Engineering Institute of Canada and the American and Canadian Society of Mechanical Engineering. He was the recipient of the Engineering Medal from Professional Engineers Ontario. He is an Associate Editor for the International Journal of Vehicle Autonomous Systems and International Journal of Powertrains.View more
