Two-step Thompson Sampling for Multi-Armed Bandit Problem

An important feature of sequential decision-making under uncertain conditions is to strike a balance between exploration (obtaining information about the value of actions) and utilization (selecting the best action based on existing knowledge). The multi-armed bandit is a classic task for balancing their relationship. As the agent has learnt the relationship between two options in a 2-armed bandit task, it will no longer explore but instead use them directly. Inspired by the research, an algorithm based on two-step Thompson sampling (T-TS) is proposed to solve multi-armed bandit problem. First, posterior Beta distributions for the reward of each arm and the probability of the superior arm were established respectively. In the first-step TS, Thompson sampling was used to extract the maximum value from the posterior distribution of probability for the superior arm, determining it as the selected arm. In the two-step TS, Thompson sampling was used to sample from the distributions of the reward. During the updating, the posterior distribution of the probability for the superior arm was updated based on observations including actions and reward, and the distribution of the reward was updated based on the reward feedback. We performed the comparative experiments to solve the static multi-bandit tasks with 10, 20, 40, and 80 arms respectively by the state-of-the-art B-UCB algorithm, Thompson sampling and the proposed T-TS. The results shown that our proposed T-TS can learn the superior action more quickly with the best robustness.