Design of Reinforcement Learning based PI controller for nonlinear Multivariable System

With the advancement of computational power, it has become easier to approximate a complex policy function using a deep neural network to achieve better accuracy and performance. Hence the policies are formulated as parameterized deep neural networks and the same are trained using state-of-the-art policy gradient algorithms. Therefore in recent years, reinforcement learning has gained much attention and its advantages make it ideal for the adaptive tuning of PI controllers. This manuscript aims to build an adaptive PI controller using Reinforcement Learning (RL) and Deep Neural Networks (DNN) for 1) nonlinear multivariable systems and 2) non-minimum phase multivariable systems. The main objective is to design a PI controller using RL formulation for nonlinear processes, where nonlinear dynamics may be unknown, by training the agents through interactions with linearized models. Proximal Policy Optimization (PPO) algorithm with dynamic action space is proposed in this manuscript. The proposed PPO based controllers achieve better performance and provide smoother convergence compared to the existing RL based controller. The effectiveness of the proposed RL-based PI controller is demonstrated through a Quadruple-Tank system with nonlinear dynamic behavior. The system exhibits both minimum and non-minimum characteristics based on the multivariable zero location. The reward function is also adapted to aid the control applications and leads to efficient learning and convergence. The results show that the controller performed well on the actual process in both minimum and non-minimum phase cases.