Abstract:
Federated Learning (FL) is a distributed machine learning framework in parallel and distributed systems. However, the systems’ Non-Independent and Identically Distributed...Show MoreMetadata
Abstract:
Federated Learning (FL) is a distributed machine learning framework in parallel and distributed systems. However, the systems’ Non-Independent and Identically Distributed (Non-IID) data negatively affect the communication efficiency, since clients with different datasets may cause significant gaps to the local gradients in each communication round. In this article, we propose a Federated Vectorized Averaging (FedVeca) method to optimize the FL communication system on Non-IID data. Specifically, we set a novel objective for the global model which is related to the local gradients. The local gradient is defined as a bi-directional vector with step size and direction, where the step size is the number of local updates and the direction is divided into positive and negative according to our definition. In FedVeca, the direction is influenced by the step size, thus we average the bi-directional vectors to reduce the effect of different step sizes. Then, we theoretically analyze the relationship between the step sizes and the global objective, and obtain upper bounds on the step sizes per communication round. Based on the upper bounds, we design an algorithm for the server and the client to adaptively adjusts the step sizes that make the objective close to the optimum. Finally, we conduct experiments on different datasets, models and scenarios by building a prototype system, and the experimental results demonstrate the effectiveness and efficiency of the FedVeca method.
Published in: IEEE Transactions on Parallel and Distributed Systems ( Volume: 35, Issue: 11, November 2024)
Funding Agency:
Hainan Blockchain Technology Engineering Research Center, School of Computer Science and Technology, Hainan University, Haikou, China
Ping Luo (Student Member, IEEE) received the MAEng degree in computer science and technology from Hainan University, Haikou, China. He is currently working toward the PhD degree in computer science and technology from the National University of Defense Technology (NUDT), Changsha, China. His research interests include convex optimization, the industrial Internet of Things and artificial intelligence.
Ping Luo (Student Member, IEEE) received the MAEng degree in computer science and technology from Hainan University, Haikou, China. He is currently working toward the PhD degree in computer science and technology from the National University of Defense Technology (NUDT), Changsha, China. His research interests include convex optimization, the industrial Internet of Things and artificial intelligence.View more
Hainan Blockchain Technology Engineering Research Center, School of Computer Science and Technology, Hainan University, Haikou, China
Jieren Cheng (Member, IEEE) received the PhD degree in computer science and technology from the National University of Defense Technology (NUDT), in 2010. He is now a professor and the associate dean of the School of Computer Science & Technology, Hainan University, China. He is awarded as “Famous South China Sea Scholar”. He serves as the director of the Hainan Provincial Blockchain Technology Engineering Research Center...Show More
Jieren Cheng (Member, IEEE) received the PhD degree in computer science and technology from the National University of Defense Technology (NUDT), in 2010. He is now a professor and the associate dean of the School of Computer Science & Technology, Hainan University, China. He is awarded as “Famous South China Sea Scholar”. He serves as the director of the Hainan Provincial Blockchain Technology Engineering Research Center...View more
Hainan Blockchain Technology Engineering Research Center, School of Computer Science and Technology, Hainan University, Haikou, China
N. Xiong (Senior Member, IEEE) received the PhD degree from the School of Information Science, Japan Advanced Institute of Science and Technology (JAIST), in 2008. He is current a distinguished professor with National Engineering Research Center for E-Learning, Central China Normal University (CCNU), Wuhan, Hu Bei Province, 430079, China. He is also with the Department of Computer Science, Georgia State University, Atlant...Show More
N. Xiong (Senior Member, IEEE) received the PhD degree from the School of Information Science, Japan Advanced Institute of Science and Technology (JAIST), in 2008. He is current a distinguished professor with National Engineering Research Center for E-Learning, Central China Normal University (CCNU), Wuhan, Hu Bei Province, 430079, China. He is also with the Department of Computer Science, Georgia State University, Atlant...View more
Hainan Blockchain Technology Engineering Research Center, School of Computer Science and Technology, Hainan University, Haikou, China
Zhenhao Liu (Student Member, IEEE) received the MAEng degree in electronic information from Hainan University, Haikou, China. He is currently working toward the PhD degree in agricultural information engineering with Huazhong Agricultural University, Wuhan, China. His research interests include federated learning, privacy-preserving, and artificial intelligence.
Zhenhao Liu (Student Member, IEEE) received the MAEng degree in electronic information from Hainan University, Haikou, China. He is currently working toward the PhD degree in agricultural information engineering with Huazhong Agricultural University, Wuhan, China. His research interests include federated learning, privacy-preserving, and artificial intelligence.View more
Department of Computer and Information Sciences, Temple University, Philadelphia, PA, USA
Jie Wu (Fellow, IEEE) is the director of the Center for Networked Computing and Laura H. Carnell professor with Temple University. He also serves as the director of International Affairs with the College of Science and Technology. He served as chair of the Department of Computer and Information Sciences from the summer of 2009 to the summer of 2016 and associate vice provost for International Affairs from the fall of 2015...Show More
Jie Wu (Fellow, IEEE) is the director of the Center for Networked Computing and Laura H. Carnell professor with Temple University. He also serves as the director of International Affairs with the College of Science and Technology. He served as chair of the Department of Computer and Information Sciences from the summer of 2009 to the summer of 2016 and associate vice provost for International Affairs from the fall of 2015...View more
Contents Hainan Blockchain Technology Engineering Research Center, School of Computer Science and Technology, Hainan University, Haikou, China
Ping Luo (Student Member, IEEE) received the MAEng degree in computer science and technology from Hainan University, Haikou, China. He is currently working toward the PhD degree in computer science and technology from the National University of Defense Technology (NUDT), Changsha, China. His research interests include convex optimization, the industrial Internet of Things and artificial intelligence.
Ping Luo (Student Member, IEEE) received the MAEng degree in computer science and technology from Hainan University, Haikou, China. He is currently working toward the PhD degree in computer science and technology from the National University of Defense Technology (NUDT), Changsha, China. His research interests include convex optimization, the industrial Internet of Things and artificial intelligence.View more
Hainan Blockchain Technology Engineering Research Center, School of Computer Science and Technology, Hainan University, Haikou, China
Jieren Cheng (Member, IEEE) received the PhD degree in computer science and technology from the National University of Defense Technology (NUDT), in 2010. He is now a professor and the associate dean of the School of Computer Science & Technology, Hainan University, China. He is awarded as “Famous South China Sea Scholar”. He serves as the director of the Hainan Provincial Blockchain Technology Engineering Research Center. His research interests include blockchain, Big Data, cloud computing, cyberecurity, artificial intelligence and intelligent transportation.
Jieren Cheng (Member, IEEE) received the PhD degree in computer science and technology from the National University of Defense Technology (NUDT), in 2010. He is now a professor and the associate dean of the School of Computer Science & Technology, Hainan University, China. He is awarded as “Famous South China Sea Scholar”. He serves as the director of the Hainan Provincial Blockchain Technology Engineering Research Center. His research interests include blockchain, Big Data, cloud computing, cyberecurity, artificial intelligence and intelligent transportation.View more
Hainan Blockchain Technology Engineering Research Center, School of Computer Science and Technology, Hainan University, Haikou, China
N. Xiong (Senior Member, IEEE) received the PhD degree from the School of Information Science, Japan Advanced Institute of Science and Technology (JAIST), in 2008. He is current a distinguished professor with National Engineering Research Center for E-Learning, Central China Normal University (CCNU), Wuhan, Hu Bei Province, 430079, China. He is also with the Department of Computer Science, Georgia State University, Atlanta, Georgia. His research interests include deep learning, reliable networks, software engineering, and Big Data analytics. He works in CCNU for many years, and obtained many research funding and many industrial projects. He published more than 600 journal paper with more than 200 IEEE journal papers. He also creates a company about design and analysis for complex reliable software systems, and obtains more than 10 patents.
N. Xiong (Senior Member, IEEE) received the PhD degree from the School of Information Science, Japan Advanced Institute of Science and Technology (JAIST), in 2008. He is current a distinguished professor with National Engineering Research Center for E-Learning, Central China Normal University (CCNU), Wuhan, Hu Bei Province, 430079, China. He is also with the Department of Computer Science, Georgia State University, Atlanta, Georgia. His research interests include deep learning, reliable networks, software engineering, and Big Data analytics. He works in CCNU for many years, and obtained many research funding and many industrial projects. He published more than 600 journal paper with more than 200 IEEE journal papers. He also creates a company about design and analysis for complex reliable software systems, and obtains more than 10 patents.View more
Hainan Blockchain Technology Engineering Research Center, School of Computer Science and Technology, Hainan University, Haikou, China
Zhenhao Liu (Student Member, IEEE) received the MAEng degree in electronic information from Hainan University, Haikou, China. He is currently working toward the PhD degree in agricultural information engineering with Huazhong Agricultural University, Wuhan, China. His research interests include federated learning, privacy-preserving, and artificial intelligence.
Zhenhao Liu (Student Member, IEEE) received the MAEng degree in electronic information from Hainan University, Haikou, China. He is currently working toward the PhD degree in agricultural information engineering with Huazhong Agricultural University, Wuhan, China. His research interests include federated learning, privacy-preserving, and artificial intelligence.View more
Department of Computer and Information Sciences, Temple University, Philadelphia, PA, USA
Jie Wu (Fellow, IEEE) is the director of the Center for Networked Computing and Laura H. Carnell professor with Temple University. He also serves as the director of International Affairs with the College of Science and Technology. He served as chair of the Department of Computer and Information Sciences from the summer of 2009 to the summer of 2016 and associate vice provost for International Affairs from the fall of 2015 to the summer of 2017. Prior to joining Temple University, he was a program director with the National Science Foundation and was a distinguished professor with Florida Atlantic University. His current research interests include mobile computing and wireless networks, routing protocols, network trust and security, distributed algorithms, applied machine learning, and cloud computing. He regularly publishes in scholarly journals, conference proceedings, and books. He serves on several editorial boards, including IEEE Transactions on Mobile Computing, IEEE Transactions on Service Computing, Journal of Parallel and Distributed Computing, and Journal of Computer Science and Technology. He is/was general chair/co-chair for IEEE IPDPS’08, IEEE DCOSS’09, IEEE ICDCS’13, ACM MobiHoc’14, ICPP’16, IEEE CNS’16, WiOpt’21, and ICDCN’22 as well as program chair/cochair for IEEE MASS’04, IEEE INFO- COM’11, CCF CNCC’13, and ICCCN’20. He was an IEEE Computer Society Distinguished Visitor, ACM Distinguished Speaker, and chair for the IEEE Technical Committee on Distributed Processing (TCDP). He is a fellow of the AAAS. He is the recipient of the 2011 China Computer Federation (CCF) Overseas Outstanding Achievement Award.
Jie Wu (Fellow, IEEE) is the director of the Center for Networked Computing and Laura H. Carnell professor with Temple University. He also serves as the director of International Affairs with the College of Science and Technology. He served as chair of the Department of Computer and Information Sciences from the summer of 2009 to the summer of 2016 and associate vice provost for International Affairs from the fall of 2015 to the summer of 2017. Prior to joining Temple University, he was a program director with the National Science Foundation and was a distinguished professor with Florida Atlantic University. His current research interests include mobile computing and wireless networks, routing protocols, network trust and security, distributed algorithms, applied machine learning, and cloud computing. He regularly publishes in scholarly journals, conference proceedings, and books. He serves on several editorial boards, including IEEE Transactions on Mobile Computing, IEEE Transactions on Service Computing, Journal of Parallel and Distributed Computing, and Journal of Computer Science and Technology. He is/was general chair/co-chair for IEEE IPDPS’08, IEEE DCOSS’09, IEEE ICDCS’13, ACM MobiHoc’14, ICPP’16, IEEE CNS’16, WiOpt’21, and ICDCN’22 as well as program chair/cochair for IEEE MASS’04, IEEE INFO- COM’11, CCF CNCC’13, and ICCCN’20. He was an IEEE Computer Society Distinguished Visitor, ACM Distinguished Speaker, and chair for the IEEE Technical Committee on Distributed Processing (TCDP). He is a fellow of the AAAS. He is the recipient of the 2011 China Computer Federation (CCF) Overseas Outstanding Achievement Award.View more
