A TDRL Model for the Emotion of Regret

To better understand the nature, function and elicitation conditions of emotion it is important to approach studying emotion from a multidisciplinary perspective involving psychology, neuroscience and affective computing. Recently, the TDRL Theory of Emotion has been proposed. It defines emotions as variations of temporal difference assessments in reinforcement learning. In this paper we present new evidence for this theory. We show that regret - a negative emotion that signifies that an alternative action should have been taken given new outcome evidence - is modelled by a particular form of TD error assessment. In our model regret is attributed to each action in the state-action trace of an agent for which - after new reward evidence - an alternative action becomes the best action in that state (the new argmax) after adjusting the action value of the chosen action in that state. Regret intensity is modeled as the difference between this new best action and the adjusted old best action, reflecting the additional amount of return that could have been received should that alternative have been chose. We show in simulation experiments how regret varies depending on the amount of adjustment as well as the adjustment mechanism, i.e. Q-trace, Sarsa-trace, and Monte Carlo (MC) re-evaluation of action values. Our work shows plausible regret attribution to actions, when this model of regret is coupled with MC action value update. This is important evidence that regret can be seen as a particular variation of TD error assessment involving counterfactual thinking.