Modelling Grid Navigation Using Reinforcement Learning Linear Ballistic Accumulators

Reinforcement Learning (RL) models constitute an important subset of models used in studying many facets of human learning including Motor Sequence Learning. However, conventional action selection in RL models such as softmax-based choice rules lack biological plausibility and do not offer mechanistic explanations. Furthermore, they also do not use response time data in model fitting, which can be indicative of the difference in value of alternate choices as perceived by subjects. Evidence Accumulation Models(EAM) such as Linear Ballistic Accumulators(LBA) provide a solution to many of the above problems. In this study, we use RL algorithms integrated with an LBA model to model human behaviour in the Grid-Sailing Task. The task involves navigating a grid to reach a goal position where the participant can choose from three possible actions. We fit RLLBA models using three different RL algorithms: a model using only Model Based updates, and two models that arbitrates between Model Based and Model Free learning, where Weight Based Arbitration is used in one and Value of Information(Vol) based Arbitration is used in the other. When following the actions of human subjects, we find a significant negative correlation between the Variance in Q-values and Response Time, motivating a competition based mechanism of action selection. When the three models are fit to data we find that VoI based models provide the best fit to data. We find that such models are able to predict actions made by human participants with accuracy comparable to that of conventional RL models and also predict the response times taken by the subject within reasonable margins. We discuss the implications of these results and outline scenarios where it would be advantageous to use RLEAM especially in the domain of Motor Learning.