Self-improving Reactive Agents: Case Studies of Reinforcement Learning Frameworks

The purpose of this work: is to investigate and evaluate different reinforcement learning frameworks using connectionist networks. I study four frameworks, which are adopted from the ideas developed in [Barto. Sutton&Walkins, 1989; Walkins. 1989; Sutton. 19901. The four frameworks are based on two learning procedures: the Temporal Difference methods for solving the credit assignment problem. and the backpropagation algorithm for developing appropriate internal representations. Two of them also involve learning a world model and using it to speed learning. To evaluate their performance, I design a dynamic environment and implement different learning agents. using the different frameworks. to survive in it, The environment is nontrivial and nondeterministic. Surprisingly. all of the agents can learn to survive fairly well in a reasonable time frame. This paper describes the learning agents and their performance. and summarizes the learning algorithms and the lessons I learned from this study.