Application of Neural Network Controller and Policy Gradient Reinforcement Learning on Modular Multilevel Converter (MMC) - a Proof of Concept

Nowadays deep reinforcement learning method has been introduced to power electronic converter control, but the current applications are relatively simple since the converter needing to be controlled has only a few variables. Applying the deep reinforcement learning method on the complicated converter with more variables has not been seen yet. This paper shows that the reinforcement learning also has potential to control a complicated modular multilevel converter (MMC). In our method, the capacitor voltages of submodules (SMs), arm currents and output currents are directly sent to a neural network controller, and the outputs of the controller are used as the gating signals of the SMs directly. The Policy Gradient (PG) based reinforcement learning algorithm is used without knowing any model information of the MMC. After learning, all the variables in the MMC can be well controlled, i.e., the output current can track to the reference, the capacitor voltage of SMs can be well balanced, the upper and lower arm voltages are well balanced. The feasibility of the proposed reinforcement learning controller is verified in Python and PLECS.