Abstract:
This article investigates the problem of communication-efficient and resilient multiagent reinforcement learning (MARL). Specifically, we consider a setting where a set o...Show MoreMetadata
Abstract:
This article investigates the problem of communication-efficient and resilient multiagent reinforcement learning (MARL). Specifically, we consider a setting where a set of agents are interconnected over a given network, and can only exchange information with their neighbors. Each agent observes a common Markov Decision Process and has a local cost which is a function of the current system state and the applied control action. The goal of MARL is for all agents to learn a policy that optimizes the infinite horizon discounted average of all their costs. Within this general setting, we consider two extensions to existing MARL algorithms. First, we provide an event-triggered learning rule where agents only exchange information with their neighbors if a certain triggering condition is satisfied. We show that this enables learning while reducing the amount of communication. Next, we consider the scenario where some of the agents can be adversarial (as captured by the Byzantine attack model), and arbitrarily deviate from the prescribed learning algorithm. We establish a fundamental trade-off between optimality and resilience when Byzantine agents are present. We then create a resilient algorithm and show almost sure convergence of all reliable agents’ value functions to the neighborhood of the optimal value function of all reliable agents, under certain conditions on the network topology. When the optimal Q -values are sufficiently separated for different actions, we show that all reliable agents can learn the optimal policy under our algorithm.
Published in: IEEE Transactions on Neural Networks and Learning Systems ( Volume: 35, Issue: 3, March 2024)
Funding Agency:
Department of Electrical Engineering, University of Texas at Arlington, Arlington, TX, USA
Yijing Xie (Member, IEEE) received the B.E. degree in automation from Wuhan University, Wuhan, China, in 2014, and the Ph.D. degree in control science and engineering from Shanghai Jiao Tong University, Shanghai, China, in 2020.
She was a Lillian Gilbreth Post-Doctoral Fellow at Purdue University, West Lafayette, IN, USA. She is an Assistant Professor at the Department of Electrical Engineering, University of Texas at Arli...Show More
Yijing Xie (Member, IEEE) received the B.E. degree in automation from Wuhan University, Wuhan, China, in 2014, and the Ph.D. degree in control science and engineering from Shanghai Jiao Tong University, Shanghai, China, in 2020.
She was a Lillian Gilbreth Post-Doctoral Fellow at Purdue University, West Lafayette, IN, USA. She is an Assistant Professor at the Department of Electrical Engineering, University of Texas at Arli...View more
School of Aeronautics and Astronautics, Purdue University, West Lafayette, IN, USA
Shaoshuai Mou (Member, IEEE) received the Ph.D. degree in electrical engineering from Yale University, New Haven, CT, USA, in 2014.
He worked as a Post-Doctoral Researcher at MIT, Cambridge, MA, USA, for a year, and then joined Purdue University, West Lafayette, IN, USA, as a Tenure-Track Assistant Professor, in August 2015. He is the Elmer Bruhn Associate Professor of Aeronautics and Astronautics at Purdue University. His...Show More
Shaoshuai Mou (Member, IEEE) received the Ph.D. degree in electrical engineering from Yale University, New Haven, CT, USA, in 2014.
He worked as a Post-Doctoral Researcher at MIT, Cambridge, MA, USA, for a year, and then joined Purdue University, West Lafayette, IN, USA, as a Tenure-Track Assistant Professor, in August 2015. He is the Elmer Bruhn Associate Professor of Aeronautics and Astronautics at Purdue University. His...View more
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA
Shreyas Sundaram (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Illinois at Urbana-Champaign, Champaign, IL, USA, in 2009.
He was a Post-Doctoral Researcher at the University of Pennsylvania, Philadelphia, PA, USA, from 2009 to 2010, and an Assistant Professor at the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada, from 2010 t...Show More
Shreyas Sundaram (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Illinois at Urbana-Champaign, Champaign, IL, USA, in 2009.
He was a Post-Doctoral Researcher at the University of Pennsylvania, Philadelphia, PA, USA, from 2009 to 2010, and an Assistant Professor at the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada, from 2010 t...View more
Contents Department of Electrical Engineering, University of Texas at Arlington, Arlington, TX, USA
Yijing Xie (Member, IEEE) received the B.E. degree in automation from Wuhan University, Wuhan, China, in 2014, and the Ph.D. degree in control science and engineering from Shanghai Jiao Tong University, Shanghai, China, in 2020.
She was a Lillian Gilbreth Post-Doctoral Fellow at Purdue University, West Lafayette, IN, USA. She is an Assistant Professor at the Department of Electrical Engineering, University of Texas at Arlington, Arlington, TX, USA. Her research focuses on control theory, multiagent systems, cyber-physical systems, and smart grids.
Yijing Xie (Member, IEEE) received the B.E. degree in automation from Wuhan University, Wuhan, China, in 2014, and the Ph.D. degree in control science and engineering from Shanghai Jiao Tong University, Shanghai, China, in 2020.
She was a Lillian Gilbreth Post-Doctoral Fellow at Purdue University, West Lafayette, IN, USA. She is an Assistant Professor at the Department of Electrical Engineering, University of Texas at Arlington, Arlington, TX, USA. Her research focuses on control theory, multiagent systems, cyber-physical systems, and smart grids.View more
School of Aeronautics and Astronautics, Purdue University, West Lafayette, IN, USA
Shaoshuai Mou (Member, IEEE) received the Ph.D. degree in electrical engineering from Yale University, New Haven, CT, USA, in 2014.
He worked as a Post-Doctoral Researcher at MIT, Cambridge, MA, USA, for a year, and then joined Purdue University, West Lafayette, IN, USA, as a Tenure-Track Assistant Professor, in August 2015. He is the Elmer Bruhn Associate Professor of Aeronautics and Astronautics at Purdue University. His research has been focusing on advancing control theories with recent progress in optimization, networks, and learning to address fundamental challenges in autonomous systems, with particular research interests in distributed control, optimization and learning, control of autonomous robots, human–robot teaming, cybersecurity, and resilience.
Shaoshuai Mou (Member, IEEE) received the Ph.D. degree in electrical engineering from Yale University, New Haven, CT, USA, in 2014.
He worked as a Post-Doctoral Researcher at MIT, Cambridge, MA, USA, for a year, and then joined Purdue University, West Lafayette, IN, USA, as a Tenure-Track Assistant Professor, in August 2015. He is the Elmer Bruhn Associate Professor of Aeronautics and Astronautics at Purdue University. His research has been focusing on advancing control theories with recent progress in optimization, networks, and learning to address fundamental challenges in autonomous systems, with particular research interests in distributed control, optimization and learning, control of autonomous robots, human–robot teaming, cybersecurity, and resilience.View more
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA
Shreyas Sundaram (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Illinois at Urbana-Champaign, Champaign, IL, USA, in 2009.
He was a Post-Doctoral Researcher at the University of Pennsylvania, Philadelphia, PA, USA, from 2009 to 2010, and an Assistant Professor at the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada, from 2010 to 2014. He is the Marie Gordon Professor at the Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA. His research interests include resilient control of networked autonomous systems, network science, optimization and learning in multiagent systems, analysis of large-scale dynamical systems, and game theory.
Dr. Sundaram was a recipient of the NSF CAREER Award. At Purdue, he has received several awards including the HKN Outstanding Professor Award, the Outstanding Mentor of Engineering Graduate Students Award, the Hesselberth Award for Teaching Excellence, and the Ruth and Joel Spira Outstanding Teacher Award.
Shreyas Sundaram (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from the University of Illinois at Urbana-Champaign, Champaign, IL, USA, in 2009.
He was a Post-Doctoral Researcher at the University of Pennsylvania, Philadelphia, PA, USA, from 2009 to 2010, and an Assistant Professor at the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada, from 2010 to 2014. He is the Marie Gordon Professor at the Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA. His research interests include resilient control of networked autonomous systems, network science, optimization and learning in multiagent systems, analysis of large-scale dynamical systems, and game theory.
Dr. Sundaram was a recipient of the NSF CAREER Award. At Purdue, he has received several awards including the HKN Outstanding Professor Award, the Outstanding Mentor of Engineering Graduate Students Award, the Hesselberth Award for Teaching Excellence, and the Ruth and Joel Spira Outstanding Teacher Award.View more
