Accuracy-Based Learning Classifier Systems for Multistep Reinforcement Learning: A Fuzzy Logic Approach to Handling Continuous Inputs and Learning Continuous Actions

Despite their proven effectiveness, many Michigan learning classifier systems (LCSs) cannot perform multistep reinforcement learning in continuous spaces. To meet this technical challenge, some LCSs have been designed to learn fuzzy logic rules. They can be largely classified into strength-based and accuracy-based systems. The latter is gaining more research attention in the last decade. However, existing accuracy-based learning systems either address primarily single-step learning problems or require the action space to be discrete. In this paper, a new accuracy-based learning fuzzy classifier system (LFCS) is developed to explicitly handle continuous state input and continuous action output during multistep reinforcement learning. Several technical improvements have been achieved while developing the new learning algorithm. Particularly, we have successfully extended Q-learning like credit assignment methods to continuous spaces. To enable direct learning of stochastic strategies for action selection, we have also proposed to use a new fuzzy logic system with stochastic action outputs. Moreover, fine-grained learning of fuzzy rules has been achieved effectively in our algorithm by using a natural gradient learning method. It is the first time that these techniques are utilized substantially in any accuracy-based LFCSs. Meanwhile, in comparison with several recently proposed learning algorithms, our algorithm is shown to perform highly competitively on four benchmark learning problems and a robotics problem. The practical usefulness of our algorithm is also demonstrated by improving the performance of a wireless body area network.

I. Introduction

Learning classifier systems (LCSs), a promising technology for evolutionary machine learning, seek to solve a learning problem by evolving a group of IF-THEN rules [31], [32]. These rules are not only easily understandable (e.g., as opposed to the neural network based approach) but also general enough for a wide range of learning tasks, such as classification [4], pattern recognition [35], and clustering [49]. Recently, the successful application of LCSs on many real-world problems has triggered increasing research attention [3], [51], [52]. Michigan LCSs in particular, which are the main focus of this paper, have been extensively studied for their remarkable ability of solving various multistep reinforcement learning problems [11], [23], [56], for example robotic control problems [2], [54], chemical reaction control problems [9], and traffic control problems [15].