Autonomous Shaping by High Density Visited States

Shaping is an effective method to reduce the state space that agent has to explore. But the shaping signal usually is provided by external observer, which requires lots of efforts, and weakens agent's autonomy. We propose an approach to make agent can autonomously shape itself by high density visited states. By gathering state trajectories that agent passed in training episodes, and eliminating the state loops in these state trajectories, then agent can find the high density visited states from these acyclic state trajectories. A high density visited state means agent has high frequency to pass it when agent wants to achieve the goal. So the high density visited state can be used to shape agent's exploration and make it reaching the goal faster. The experiment results on the Maze problem illustrated our approach being very effective. The major contribution is we make agent can autonomously shape itself by its experience.