A Novel Q-Learning Approach with Continuous States and Actions

This paper presents a generalized Q-learning method termed dynamic fuzzy continuous-action Q-learning (DFCAQ) that works in continuous domains. It can be regarded as an extension of Millan's work in continuous-action Q-learning. In the DFCAQ approach, continuous states and actions are generated via a fuzzy structure. Instead of considering actions selected by the nearest unit only in the original continuous-action Q-learning, the global action is generated via a fuzzy approach. Compared with Jouffe's fuzzy Q-learning, the DFCAQ fuzzy structure can be automatically and dynamically generated. At the same time, the local actions in the DFCAQ method are average values of the discrete actions weighted by their Q-values. In addition, comparison studies in robotics domains show the superiority of the proposed DFCAQ method.