Solving Unit Commitment Problems with Multi-step Deep Reinforcement Learning

Solving the unit commitment (UC) problem in a computationally efficient manner is a critical issue of electricity market operations. Optimization-based methods such as heuristics, dynamic programming, and mixed-integer quadratic programming (MIQP) often yield good solutions to the UC problem. However, the computation time of optimization-based methods grows exponentially with the number of generating units, which is a major bottleneck in practice. To address this issue, we formulate the UC problem as a Markov decision process and propose a novel multi-step deep reinforcement learning (RL)-based algorithm to solve the problem. We approximate the action-value function with neural networks and design an algorithm to determine the feasible action space. Numerical studies on a 5-generator test case show that our proposed algorithm significantly outperforms the deep Q-learning and yields similar level of performance as that of MIQP-based optimization in terms of optimality. The computation time of our proposed algorithm is much shorter than that of MIQP-based optimization methods.