Loading [a11y]/accessibility-menu.js
A Multi-Agent Deep Reinforcement Learning-Based Handover Scheme for Mega-Constellation Under Dynamic Propagation Conditions | IEEE Journals & Magazine | IEEE Xplore

A Multi-Agent Deep Reinforcement Learning-Based Handover Scheme for Mega-Constellation Under Dynamic Propagation Conditions


Abstract:

With the rapidly increasing number of satellites, the handover scheme design is critically important for the low Earth orbit (LEO) satellite networks, especially for the ...Show More

Abstract:

With the rapidly increasing number of satellites, the handover scheme design is critically important for the low Earth orbit (LEO) satellite networks, especially for the mega-constellations that include massive number of LEO satellites. However, the existing handover schemes for LEO satellite networks are designed based on the static propagation conditions, which cannot satisfy the dynamic feature of communication environment caused by the mobility of LEO satellites and users. To address this issue, a centralized adaptive intelligent handover scheme for mega-constellations is proposed, where the dynamics of the propagation conditions and limited LEO satellite capacity are taken into considerations. Specifically, we first use a three-state Markov model to characterize the dynamically varying propagation conditions between satellites and users. Then, the Loo model is employed to describe the dynamic land mobile satellite channels. By considering the user transmission rate requirement and the load-balancing demand of satellites, we design the user utility function and formulate an optimization problem that aims to maximize the overall long-term utility of the network. To reduce the handover decision-making complexity, a multi-agent successive hysteretic deep Q-learning algorithm is developed and it can efficiently solve the formulated problem by reducing the state and action space. To reduce the signaling overhead and the computation complexity of the proposed centralized handover scheme brought to the control center, a distributed intelligent handover scheme is further developed, where each user is enabled to independently make the handover decision only based on the local information. Simulation results show that both the proposed centralized and distributed approaches can efficiently improve the network performance over the existing schemes.
Published in: IEEE Transactions on Wireless Communications ( Volume: 23, Issue: 10, October 2024)
Page(s): 13579 - 13596
Date of Publication: 06 June 2024

ISSN Information:

Funding Agency:

Author image of Haotian Liu
School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
Haotian Liu received the B.S. degree in information engineering from Xi’an Jiaotong University, China, in 2021, where he is currently pursuing the M.S. degree in information and communications engineering. His research interests include large-scale LEO satellite communication, satellite handover, beam hopping, resource allocation, and reinforcement learning.
Haotian Liu received the B.S. degree in information engineering from Xi’an Jiaotong University, China, in 2021, where he is currently pursuing the M.S. degree in information and communications engineering. His research interests include large-scale LEO satellite communication, satellite handover, beam hopping, resource allocation, and reinforcement learning.View more
Author image of Yichen Wang
School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
Yichen Wang (Member, IEEE) received the B.S. degree in information engineering and the Ph.D. degree in information and communications engineering from Xi’an Jiaotong University, China, in 2007 and 2013, respectively.
From August 2014 to August 2015, he was a Visiting Scholar with the Signal and Information Group, Department of Electrical and Computer Engineering, University of Maryland, College Park, USA. He is currently a...Show More
Yichen Wang (Member, IEEE) received the B.S. degree in information engineering and the Ph.D. degree in information and communications engineering from Xi’an Jiaotong University, China, in 2007 and 2013, respectively.
From August 2014 to August 2015, he was a Visiting Scholar with the Signal and Information Group, Department of Electrical and Computer Engineering, University of Maryland, College Park, USA. He is currently a...View more
Author image of Peixuan Li
School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
Peixuan Li received the B.S. degree in information engineering from Xi’an Jiaotong University, China, in 2022, where she is currently pursuing the M.S. degree in information and communications engineering. Her research interests include large-scale LEO satellite communication, resource allocation, edge-cloud collaboration, and semantic communication.
Peixuan Li received the B.S. degree in information engineering from Xi’an Jiaotong University, China, in 2022, where she is currently pursuing the M.S. degree in information and communications engineering. Her research interests include large-scale LEO satellite communication, resource allocation, edge-cloud collaboration, and semantic communication.View more
Author image of Julian Cheng
School of Engineering, The University of British Columbia, Kelowna, BC, Canada
Julian Cheng (Fellow, IEEE) received the B.Eng. degree (Hons.) in electrical engineering from the University of Victoria, Victoria, BC, Canada, in 1995, the M.Sc. (Eng.) degree in mathematics and engineering from Queen’s University, Kingston, ON, Canada, in 1997, and the Ph.D. degree in electrical engineering from the University of Alberta, Edmonton, AB, Canada, in 2003. He was with Bell Northern Research and NORTEL Netwo...Show More
Julian Cheng (Fellow, IEEE) received the B.Eng. degree (Hons.) in electrical engineering from the University of Victoria, Victoria, BC, Canada, in 1995, the M.Sc. (Eng.) degree in mathematics and engineering from Queen’s University, Kingston, ON, Canada, in 1997, and the Ph.D. degree in electrical engineering from the University of Alberta, Edmonton, AB, Canada, in 2003. He was with Bell Northern Research and NORTEL Netwo...View more

Author image of Haotian Liu
School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
Haotian Liu received the B.S. degree in information engineering from Xi’an Jiaotong University, China, in 2021, where he is currently pursuing the M.S. degree in information and communications engineering. His research interests include large-scale LEO satellite communication, satellite handover, beam hopping, resource allocation, and reinforcement learning.
Haotian Liu received the B.S. degree in information engineering from Xi’an Jiaotong University, China, in 2021, where he is currently pursuing the M.S. degree in information and communications engineering. His research interests include large-scale LEO satellite communication, satellite handover, beam hopping, resource allocation, and reinforcement learning.View more
Author image of Yichen Wang
School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
Yichen Wang (Member, IEEE) received the B.S. degree in information engineering and the Ph.D. degree in information and communications engineering from Xi’an Jiaotong University, China, in 2007 and 2013, respectively.
From August 2014 to August 2015, he was a Visiting Scholar with the Signal and Information Group, Department of Electrical and Computer Engineering, University of Maryland, College Park, USA. He is currently a Professor with the Information and Communications Engineering School and the Vice Director of the Institute of Wireless Communications and Shaanxi Smart Network and Ubiquitous Access Center, Xi’an Jiaotong University. He has published more than 100 technical papers in international journals and conferences. His current research interests include 5G/B5G/6G technologies, random access for massive networks, ultra-reliable low-latency communications, security-related technologies for wireless networks, integrated space and terrestrial information networks, statistical QoS provisioning technology for wireless networks, and resource allocation over wireless networks. He is a member of the IEEE Communications Society and the IEEE Vehicular Technology Society. He also serves and has served as the Technical Program Committee Member for many world-renowned conferences, including IEEE GLOBECOM, ICC, WCNC, VTC, and PIMRC. He received the Best Paper Award from ICCCS in 2023, the Best Letter Award from IEICE Communications Society in 2010, and the Exemplary Reviewer Award from IEEE Communications Letters in 2014. He served as the TPC Co-Chair for the IEEE VTC’16-Spring Workshop on User-Centric Networking for 5G and Beyond and the Track Co-Chair for the Cloud Communications and Networking of CHINACOM’17. He is currently serving as an Editor for KSII Transactions on Internet and Information Systems.
Yichen Wang (Member, IEEE) received the B.S. degree in information engineering and the Ph.D. degree in information and communications engineering from Xi’an Jiaotong University, China, in 2007 and 2013, respectively.
From August 2014 to August 2015, he was a Visiting Scholar with the Signal and Information Group, Department of Electrical and Computer Engineering, University of Maryland, College Park, USA. He is currently a Professor with the Information and Communications Engineering School and the Vice Director of the Institute of Wireless Communications and Shaanxi Smart Network and Ubiquitous Access Center, Xi’an Jiaotong University. He has published more than 100 technical papers in international journals and conferences. His current research interests include 5G/B5G/6G technologies, random access for massive networks, ultra-reliable low-latency communications, security-related technologies for wireless networks, integrated space and terrestrial information networks, statistical QoS provisioning technology for wireless networks, and resource allocation over wireless networks. He is a member of the IEEE Communications Society and the IEEE Vehicular Technology Society. He also serves and has served as the Technical Program Committee Member for many world-renowned conferences, including IEEE GLOBECOM, ICC, WCNC, VTC, and PIMRC. He received the Best Paper Award from ICCCS in 2023, the Best Letter Award from IEICE Communications Society in 2010, and the Exemplary Reviewer Award from IEEE Communications Letters in 2014. He served as the TPC Co-Chair for the IEEE VTC’16-Spring Workshop on User-Centric Networking for 5G and Beyond and the Track Co-Chair for the Cloud Communications and Networking of CHINACOM’17. He is currently serving as an Editor for KSII Transactions on Internet and Information Systems.View more
Author image of Peixuan Li
School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
Peixuan Li received the B.S. degree in information engineering from Xi’an Jiaotong University, China, in 2022, where she is currently pursuing the M.S. degree in information and communications engineering. Her research interests include large-scale LEO satellite communication, resource allocation, edge-cloud collaboration, and semantic communication.
Peixuan Li received the B.S. degree in information engineering from Xi’an Jiaotong University, China, in 2022, where she is currently pursuing the M.S. degree in information and communications engineering. Her research interests include large-scale LEO satellite communication, resource allocation, edge-cloud collaboration, and semantic communication.View more
Author image of Julian Cheng
School of Engineering, The University of British Columbia, Kelowna, BC, Canada
Julian Cheng (Fellow, IEEE) received the B.Eng. degree (Hons.) in electrical engineering from the University of Victoria, Victoria, BC, Canada, in 1995, the M.Sc. (Eng.) degree in mathematics and engineering from Queen’s University, Kingston, ON, Canada, in 1997, and the Ph.D. degree in electrical engineering from the University of Alberta, Edmonton, AB, Canada, in 2003. He was with Bell Northern Research and NORTEL Networks. He is currently a Full Professor with the School of Engineering, Faculty of Applied Science, The University of British Columbia, Kelowna, BC, Canada. His research interests include machine learning and deep learning for wireless communications, wireless optical technology, and quantum communications. He is a Registered Professional Engineer in British Columbia, Canada. He served as the President for Canadian Society of Information Theory from 2017 to 2021. He was the Co-Chair of the 12th Canadian Workshop on Information Theory in 2011, the 28th Biennial Symposium on Communications in 2016, and the General Co-Chair of the 2021 and 2024 IEEE Communication Theory Workshop. He is the Chair of the Radio Communications Technical Committee of the IEEE Communications Society. He was a past Associate Editor of IEEE Transactions on Communications, IEEE Transactions on Wireless Communications, IEEE Communications Letters, and IEEE Access; and an Area Editor of IEEE Transactions on Communications. He served as a Guest Editor for the Special Issues on Optical Wireless Communications and Positioning and Sensing Over Wireless Networks of IEEE Journal on Selected Areas in Communications.
Julian Cheng (Fellow, IEEE) received the B.Eng. degree (Hons.) in electrical engineering from the University of Victoria, Victoria, BC, Canada, in 1995, the M.Sc. (Eng.) degree in mathematics and engineering from Queen’s University, Kingston, ON, Canada, in 1997, and the Ph.D. degree in electrical engineering from the University of Alberta, Edmonton, AB, Canada, in 2003. He was with Bell Northern Research and NORTEL Networks. He is currently a Full Professor with the School of Engineering, Faculty of Applied Science, The University of British Columbia, Kelowna, BC, Canada. His research interests include machine learning and deep learning for wireless communications, wireless optical technology, and quantum communications. He is a Registered Professional Engineer in British Columbia, Canada. He served as the President for Canadian Society of Information Theory from 2017 to 2021. He was the Co-Chair of the 12th Canadian Workshop on Information Theory in 2011, the 28th Biennial Symposium on Communications in 2016, and the General Co-Chair of the 2021 and 2024 IEEE Communication Theory Workshop. He is the Chair of the Radio Communications Technical Committee of the IEEE Communications Society. He was a past Associate Editor of IEEE Transactions on Communications, IEEE Transactions on Wireless Communications, IEEE Communications Letters, and IEEE Access; and an Area Editor of IEEE Transactions on Communications. He served as a Guest Editor for the Special Issues on Optical Wireless Communications and Positioning and Sensing Over Wireless Networks of IEEE Journal on Selected Areas in Communications.View more

Contact IEEE to Subscribe

References

References is not available for this document.