Sharing of Energy Among Cooperative Households Using Distributed Multi-Agent Reinforcement Learning

Due to the increase of the complexity and uncertainty in the future sustainable energy system new control algorithms for decentralized acting energy entities are needed. We present an approach of distributed Reinforcement Learning in a multi-agent setup to find a control strategy of two cooperative agents within an energy cell. In order to practice energy sharing to decrease the energy cell's overall interdependence on the electrical grid, we train two independently learning agents, an energy storage and an electric power generator using Q-learning. We compare the learned strategy of the agents under partial and full observability of the environment and evaluate the interdependence of the energy cell on the electrical grid. Our results show that distributed Q-learning with independently learning agents works in the setup of an energy cell without the necessity of information exchange between agents. The algorithm under partial observability of the environment reaches comparable performance to that of full observability with fewer need of communication but at the cost of five times longer training time.

I. Introduction

The energy transition towards a more sustainable, secure and affordable energy supply (the German Energiewende[1]) consisting of a high share of renewable energy sources (RES) increases the energy system's complexity. It creates an energy system in a more decentralized pattern with many more participants. Single households may be part of the energy system not only consuming electricity but also becoming a prosumer by installing rooftop photo-voltaic (PV) systems. RES such as PV and wind power plants are intermittent in their energy generation. With increasing share of RES the challenge of projectable energy generation exacerbates. In a future energy system with many decentralized generation units it is desirable to level out energy supply and demand on a local level. The more complex the energy system becomes, the harder it is to control such a system. New control algorithms are needed to account for such a complex environment. Reinforcement Learning (RL), as a proven approach to handle those dynamic uncertain systems, have already found its way to improve sequential decision-making in several domains such as robotics and self-driving cars [2]. In literature several research works have been conducted concerning single-agent RL in the energy domain such as improving the decision-making in microgrid control [3]. A multi-agent setup of Reinforcement Learning in the energy domain is rarely found in literature.