Addressing Non-Stationarity with Relaxed f-Discounted-Sliding-Window Thompson Sampling

Multi-armed bandit (MAB) problems serve as fundamental models for sequential decision-making under uncertainty, with applications spanning diverse domains such as online advertising, recommendation systems, and resource allocation. While the majority of classical MAB algorithms assume stationary reward distributions, real-world scenarios often exhibit concept drift, posing significant challenges for traditional policies. In this paper, we introduce the Relaxed f-Discounted-Sliding-Window Thompson Sampling (RF-DSW TS) algorithm, a novel TS variant tailored to address non-stationarity in bandit problems. By incorporating relaxed discounting and sliding window mechanisms, RF-DSW TS offers enhanced adaptability to changing reward structures, enabling it to detect abrupt changes. Through our experiment and analysis, we demonstrate the superiority of RF-DSW TS in adapting to dynamic environments in comparison to its Thompson Sampling predecessors. This paper contributes to advancing the understanding of adaptive MAB policies in dynamic contexts and lays the groundwork for further research and innovation.