Loading web-font TeX/Math/Italic
Optimization Landscape of Policy Gradient Methods for Discrete-Time Static Output Feedback | IEEE Journals & Magazine | IEEE Xplore

Optimization Landscape of Policy Gradient Methods for Discrete-Time Static Output Feedback


Abstract:

In recent times, significant advancements have been made in delving into the optimization landscape of policy gradient methods for achieving optimal control in linear tim...Show More

Abstract:

In recent times, significant advancements have been made in delving into the optimization landscape of policy gradient methods for achieving optimal control in linear time-invariant (LTI) systems. Compared with state-feedback control, output-feedback control is more prevalent since the underlying state of the system may not be fully observed in many practical settings. This article analyzes the optimization landscape inherent to policy gradient methods when applied to static output feedback (SOF) control in discrete-time LTI systems subject to quadratic cost. We begin by establishing crucial properties of the SOF cost, encompassing coercivity, L -smoothness, and M -Lipschitz continuous Hessian. Despite the absence of convexity, we leverage these properties to derive novel findings regarding convergence (and nearly dimension-free rate) to stationary points for three policy gradient methods, including the vanilla policy gradient method, the natural policy gradient method, and the Gauss–Newton method. Moreover, we provide proof that the vanilla policy gradient method exhibits linear convergence toward local minima when initialized near such minima. This article concludes by presenting numerical examples that validate our theoretical findings. These results not only characterize the performance of gradient descent for optimizing the SOF problem but also provide insights into the effectiveness of general policy gradient methods within the realm of reinforcement learning.
Published in: IEEE Transactions on Cybernetics ( Volume: 54, Issue: 6, June 2024)
Page(s): 3588 - 3601
Date of Publication: 26 October 2023

ISSN Information:

PubMed ID: 37883283

Funding Agency:

Author image of Jingliang Duan
School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, China
Department of Electrical and Computer Engineering, National University of Singapore, Queenstown, Singapore
Jingliang Duan (Member, IEEE) received the Doctoral degree in mechanical engineering from the School of Vehicle and Mobility, Tsinghua University, Beijing, China, in 2021.
In 2019, he was as a Visiting Student Researcher with the Department of Mechanical Engineering, The University of California at Berkeley, Berkeley, CA, USA. Following his Ph.D. degree, he served as a Research Fellow with the Department of Electrical and ...Show More
Jingliang Duan (Member, IEEE) received the Doctoral degree in mechanical engineering from the School of Vehicle and Mobility, Tsinghua University, Beijing, China, in 2021.
In 2019, he was as a Visiting Student Researcher with the Department of Mechanical Engineering, The University of California at Berkeley, Berkeley, CA, USA. Following his Ph.D. degree, he served as a Research Fellow with the Department of Electrical and ...View more
Author image of Jie Li
School of Vehicle and Mobility, Tsinghua University, Beijing, China
Jie Li received the B.S. degree in automotive engineering from Tsinghua University, Beijing, China, in 2018, where he is currently pursuing the Ph.D. degree with the School of Vehicle and Mobility.
His current research interests include model predictive control, adaptive dynamic programming, and robust reinforcement learning.
Jie Li received the B.S. degree in automotive engineering from Tsinghua University, Beijing, China, in 2018, where he is currently pursuing the Ph.D. degree with the School of Vehicle and Mobility.
His current research interests include model predictive control, adaptive dynamic programming, and robust reinforcement learning.View more
Author image of Xuyang Chen
Department of Electrical and Computer Engineering, National University of Singapore, Queenstown, Singapore
Xuyang Chen received the B.S. degree from the Honors College, Beihang University, Beijing, China, in 2019. He is currently pursuing the Ph.D. degree with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore.
His research interests include reinforcement learning, Markov decision process, and policy gradient methods.
Xuyang Chen received the B.S. degree from the Honors College, Beihang University, Beijing, China, in 2019. He is currently pursuing the Ph.D. degree with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore.
His research interests include reinforcement learning, Markov decision process, and policy gradient methods.View more
Author image of Kai Zhao
Department of Electrical and Computer Engineering, National University of Singapore, Queenstown, Singapore
Kai Zhao (Member, IEEE) received the Ph.D. degree in control theory and control engineering from Chongqing University, Chongqing, China, in 2019.
He was a Postdoctoral Fellow with the Department of Computer and Information Science, University of Macau, Macau, China, from 2019 to 2021. He is currently a Research Fellow with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore. H...Show More
Kai Zhao (Member, IEEE) received the Ph.D. degree in control theory and control engineering from Chongqing University, Chongqing, China, in 2019.
He was a Postdoctoral Fellow with the Department of Computer and Information Science, University of Macau, Macau, China, from 2019 to 2021. He is currently a Research Fellow with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore. H...View more
Author image of Shengbo Eben Li
School of Vehicle and Mobility, Tsinghua University, Beijing, China
Shengbo Eben Li (Senior Member, IEEE) received the M.S. and Ph.D. degrees from Tsinghua University, Beijing, China, in 2006 and 2009, respectively.
He was with Stanford University, Stanford, CA, USA; University of Michigan at Ann Arbor, Ann Arbor, MI, USA; and The University of California at Berkeley, Berkeley, CA, USA. He is currently a tenured Professor with Tsinghua University. His active research interests include inte...Show More
Shengbo Eben Li (Senior Member, IEEE) received the M.S. and Ph.D. degrees from Tsinghua University, Beijing, China, in 2006 and 2009, respectively.
He was with Stanford University, Stanford, CA, USA; University of Michigan at Ann Arbor, Ann Arbor, MI, USA; and The University of California at Berkeley, Berkeley, CA, USA. He is currently a tenured Professor with Tsinghua University. His active research interests include inte...View more
Author image of Lin Zhao
Department of Electrical and Computer Engineering, National University of Singapore, Queenstown, Singapore
Lin Zhao (Member, IEEE) received the B.S. and M.S. degrees in automatic control from the Harbin Institute of Technology, Harbin, China, in 2010 and 2012, respectively, and the M.S. degree in mathematics and the Ph.D. degree in electrical and computer engineering from The Ohio State University, Columbus, OH, USA, in 2017.
From 2018 to early 2020, he was a Research Scientist with Aptiv Pittsburgh Technology Center (currently...Show More
Lin Zhao (Member, IEEE) received the B.S. and M.S. degrees in automatic control from the Harbin Institute of Technology, Harbin, China, in 2010 and 2012, respectively, and the M.S. degree in mathematics and the Ph.D. degree in electrical and computer engineering from The Ohio State University, Columbus, OH, USA, in 2017.
From 2018 to early 2020, he was a Research Scientist with Aptiv Pittsburgh Technology Center (currently...View more

Author image of Jingliang Duan
School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, China
Department of Electrical and Computer Engineering, National University of Singapore, Queenstown, Singapore
Jingliang Duan (Member, IEEE) received the Doctoral degree in mechanical engineering from the School of Vehicle and Mobility, Tsinghua University, Beijing, China, in 2021.
In 2019, he was as a Visiting Student Researcher with the Department of Mechanical Engineering, The University of California at Berkeley, Berkeley, CA, USA. Following his Ph.D. degree, he served as a Research Fellow with the Department of Electrical and Computer Engineering, The National University of Singapore, Singapore, from 2021 to 2022. He is currently a tenured Associate Professor with the School of Mechanical Engineering, University of Science and Technology Beijing, Beijing. His research interests include reinforcement learning, optimal control, and self-driving decision making.
Jingliang Duan (Member, IEEE) received the Doctoral degree in mechanical engineering from the School of Vehicle and Mobility, Tsinghua University, Beijing, China, in 2021.
In 2019, he was as a Visiting Student Researcher with the Department of Mechanical Engineering, The University of California at Berkeley, Berkeley, CA, USA. Following his Ph.D. degree, he served as a Research Fellow with the Department of Electrical and Computer Engineering, The National University of Singapore, Singapore, from 2021 to 2022. He is currently a tenured Associate Professor with the School of Mechanical Engineering, University of Science and Technology Beijing, Beijing. His research interests include reinforcement learning, optimal control, and self-driving decision making.View more
Author image of Jie Li
School of Vehicle and Mobility, Tsinghua University, Beijing, China
Jie Li received the B.S. degree in automotive engineering from Tsinghua University, Beijing, China, in 2018, where he is currently pursuing the Ph.D. degree with the School of Vehicle and Mobility.
His current research interests include model predictive control, adaptive dynamic programming, and robust reinforcement learning.
Jie Li received the B.S. degree in automotive engineering from Tsinghua University, Beijing, China, in 2018, where he is currently pursuing the Ph.D. degree with the School of Vehicle and Mobility.
His current research interests include model predictive control, adaptive dynamic programming, and robust reinforcement learning.View more
Author image of Xuyang Chen
Department of Electrical and Computer Engineering, National University of Singapore, Queenstown, Singapore
Xuyang Chen received the B.S. degree from the Honors College, Beihang University, Beijing, China, in 2019. He is currently pursuing the Ph.D. degree with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore.
His research interests include reinforcement learning, Markov decision process, and policy gradient methods.
Xuyang Chen received the B.S. degree from the Honors College, Beihang University, Beijing, China, in 2019. He is currently pursuing the Ph.D. degree with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore.
His research interests include reinforcement learning, Markov decision process, and policy gradient methods.View more
Author image of Kai Zhao
Department of Electrical and Computer Engineering, National University of Singapore, Queenstown, Singapore
Kai Zhao (Member, IEEE) received the Ph.D. degree in control theory and control engineering from Chongqing University, Chongqing, China, in 2019.
He was a Postdoctoral Fellow with the Department of Computer and Information Science, University of Macau, Macau, China, from 2019 to 2021. He is currently a Research Fellow with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore. His research interests include adaptive control and prescribed performance control.
Kai Zhao (Member, IEEE) received the Ph.D. degree in control theory and control engineering from Chongqing University, Chongqing, China, in 2019.
He was a Postdoctoral Fellow with the Department of Computer and Information Science, University of Macau, Macau, China, from 2019 to 2021. He is currently a Research Fellow with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore. His research interests include adaptive control and prescribed performance control.View more
Author image of Shengbo Eben Li
School of Vehicle and Mobility, Tsinghua University, Beijing, China
Shengbo Eben Li (Senior Member, IEEE) received the M.S. and Ph.D. degrees from Tsinghua University, Beijing, China, in 2006 and 2009, respectively.
He was with Stanford University, Stanford, CA, USA; University of Michigan at Ann Arbor, Ann Arbor, MI, USA; and The University of California at Berkeley, Berkeley, CA, USA. He is currently a tenured Professor with Tsinghua University. His active research interests include intelligent vehicles and driver assistance, reinforcement learning and distributed control, and optimal control and estimation.
Shengbo Eben Li (Senior Member, IEEE) received the M.S. and Ph.D. degrees from Tsinghua University, Beijing, China, in 2006 and 2009, respectively.
He was with Stanford University, Stanford, CA, USA; University of Michigan at Ann Arbor, Ann Arbor, MI, USA; and The University of California at Berkeley, Berkeley, CA, USA. He is currently a tenured Professor with Tsinghua University. His active research interests include intelligent vehicles and driver assistance, reinforcement learning and distributed control, and optimal control and estimation.View more
Author image of Lin Zhao
Department of Electrical and Computer Engineering, National University of Singapore, Queenstown, Singapore
Lin Zhao (Member, IEEE) received the B.S. and M.S. degrees in automatic control from the Harbin Institute of Technology, Harbin, China, in 2010 and 2012, respectively, and the M.S. degree in mathematics and the Ph.D. degree in electrical and computer engineering from The Ohio State University, Columbus, OH, USA, in 2017.
From 2018 to early 2020, he was a Research Scientist with Aptiv Pittsburgh Technology Center (currently Motional), Pittsburgh, PA, USA. He is currently an Assistant Professor with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore. His current research focuses on control and reinforcement learning with applications in robotics.
Dr. Zhao serves on the Young Editorial Board for the Journal of Systems Science and Complexity (Springer), served as the Program Co-Chair for the 17th IEEE International Conference on Control and Automation (ICCA 2022), and as a Publicity Co-Chair for the 62nd IEEE Conference on Decision and Control (CDC 2023).
Lin Zhao (Member, IEEE) received the B.S. and M.S. degrees in automatic control from the Harbin Institute of Technology, Harbin, China, in 2010 and 2012, respectively, and the M.S. degree in mathematics and the Ph.D. degree in electrical and computer engineering from The Ohio State University, Columbus, OH, USA, in 2017.
From 2018 to early 2020, he was a Research Scientist with Aptiv Pittsburgh Technology Center (currently Motional), Pittsburgh, PA, USA. He is currently an Assistant Professor with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore. His current research focuses on control and reinforcement learning with applications in robotics.
Dr. Zhao serves on the Young Editorial Board for the Journal of Systems Science and Complexity (Springer), served as the Program Co-Chair for the 17th IEEE International Conference on Control and Automation (ICCA 2022), and as a Publicity Co-Chair for the 62nd IEEE Conference on Decision and Control (CDC 2023).View more

References

References is not available for this document.