DA-FRL: A Federated Reinforcement Learning Framework for Distributed Energy Resource Optimization based on Adaptive Exploration

The integration of distributed energy resources (DERs) into renewable grids faces challenges from volatile generation, dynamic loads, and non-independent and identically distributed (non-IID) data across decentralized nodes. Existing centralized optimization methods lack scalability and privacy safeguards, while conventional federated reinforcement learning (FRL) struggles with unstable convergence under asynchronous updates. This study proposes DA-FRL (Distributed Asynchronous Federated Reinforcement Learning), fusing Asynchronous Advantage Actor-Critic (A3C) with Adaptive Proximal Policy Optimization (APPO). Innovations include: (1) an asynchronous federated aggregation mechanism combining A3C’s distributed training with APPO’s adaptive KL-divergence regularization for robust non-IID convergence; (2) a LSTM-AEPPO probabilistic forecasting model using long short-term memory (LSTM) for temporal dependencies and adaptive exploration (AEPPO) to balance exploration-exploitation trade-offs; (3) a differential privacy-preserving gradient sparsification protocol reducing communication costs via Top-k gradient compression while securing sensitive client data. Experiments on NREL wind and CDGS photovoltaic datasets show DA-FRL reduces Winkler scores by 18.7% and 21.3%, respectively, advancing secure, scalable DER coordination under dynamic grid conditions.