Sequential $Q$ -Learning With Kalman Filtering for Multirobot Cooperative Transportation

This paper presents a modified, distributed $Q$-learning algorithm, termed as sequential $Q$-learning with Kalman filtering (SQKF), for decision making associated with multirobot cooperation. The SQKF algorithm developed here has the following characteristics. 1) The learning process is arranged in a sequential manner (i.e., the robots will not make decisions simultaneously, but in a predefined sequence) so as to promote cooperation among robots and reduce their $Q$-learning spaces. 2) A robot will not update its $Q$-values with observed global rewards. Instead, it will employ a specific Kalman filter to extract its real local reward from the global reward, thereby updating its $Q$-table with this local reward. The new SQKF algorithm is intended to solve two problems in multirobot $Q$ -learning: credit assignment and behavior conflicts. The detailed procedure of the SQKF algorithm is presented, and its application is illustrated using a prototype multirobot experimental system. The experimental results show that the algorithm has better performance than the conventional single-agent $Q$-learning algorithm or the team $Q$-learning algorithm in the multirobot domain.