Safe inclusion of information about rates of variation in a reinforcement learning algorithm

There is a need to enhance reinforcement learning techniques by using prior knowledge built into the agent at its inception. The information crudeness upon which those algorithms operate may be interesting from a theoretical point of view, but large scale problems are made too difficult and unrealistic by considering the learning agent as a 'tabula rasa'. None the less, knowledge must be embedded in such a way that the structural, well-studied characteristics of the fundamental algorithms are maintained. A more general formulation of a classical reinforcement learning method is investigated. It allows for a spreading of information derived from single updates towards a neighbourhood of the instantly visited state, and converges to optimality. We show how this new formulation can be used as a mechanism to safely embed prior knowledge about expected rates of variation of action values, and practical studies demonstrate an application of the proposed algorithm.