Anxious Learning in Real-Time Heuristic Search

Real-time heuristic search methods are used when planning time available per agent's move is severely limited (e.g., while pathfinding in a video game). Such agents interleave planning and plan execution. As the agent has to move before a complete plan is computed, it is prone to be misguided by inaccuracies in its heuristic. To get out of heuristic depressions, such agents update their heuristic over time. The usual update process requires multiple state revisits which can make the agent appear irrational to the player. To alleviate such map "scrubbing" we propose a new learning mechanism inspired by the psychological notion of anxiety. Our agent maintains a level of anxiety which increases due to state revisits and decays naturally over time. Agent's anxiety causes it to update the heuristic more aggressively thus filling heuristic depressions quicker. Such anxiety-accelerated learning can be used on top of other real-time heuristic search techniques. Empirical evaluation on video-game pathfinding benchmarks demonstrates benefits for the average solution quality when the new mechanism is used by itself or in combination with expendable state marking.