Optimal Secure Control for Cyber-Physical Systems Under False Data Injection Attacks via Incremental Iterative Q-Learning Algorithm

An incremental iterative Q-learning algorithm (IIQLA) is proposed to tackle the optimal secure control problem for cyber-physical systems under false data injection attacks. Within a zero-sum game framework, the secure control problem is transformed into solving an iterative algebraic Riccati equation. To derive the optimal secure control policy from the equation, the IIQLA is developed, which does not require prior knowledge of the system dynamics. This algorithm utilizes two auxiliary variables to separate the behavior policy and the target policy, thereby improving the exploration of data. As a result, the devised control policies are more conservative, leading to solutions that are closer to the optimal policy. Moreover, by introducing an adaptive learning rate, the proposed IIQLA can accelerate the convergence speed and decrease the number of required iterations, thus alleviating the computation burden. The convergence of the proposed IIQLA with different learning rates is also analyzed. In addition, the closed-loop system is guaranteed to be asymptotically stable, and the exploration noise does not introduce bias into the optimal policies. Finally, a simulation example is provided to demonstrate the effectiveness of the proposed approach. Note to Practitioners—This study addresses the secure control challenges posed by cyber-physical systems in the presence of false data injection attacks. Traditional secure control methods mainly compensate for the systems after they are attacked, which is passive rather than active resistance to attacks. In the zero-sum game framework, an IIQLA is proposed to obtain the optimal secure control policy without the system dynamics, which integrates an improved adaptive relaxation function and more history information. The IIQLA helps accelerate the convergence speed, decrease the number of required iterations, and improve the exploration of data. The designed method is well suited for real systems that face unpredictable events, such as cyber-attacks, disturbances, and faults.