Offline Deep Reinforcement Learning Two-stage Optimization Framework Applied to Recommendation Systems

Most traditional recommendation systems (RS) improve the model structure to obtain valid user behavior representation and make better recommendations. However, limited by the scale of the model, the model cannot accommodate an extended range of user historical data and cannot effectively extract long-term preferences. Reinforcement learning (RL) aims to maximize the cumulative return. Introducing RL can smoothly solve the long-term and short-term profit balance. However, there are specific problems with the direct combination of the two. Training RS that relies on offline user data will cause the strategy to suffer from an overestimation of action and distribution shift. Unlike the traditional RL environment, the ample space of the RS will also cause problems of insufficient exploration and sparse rewards, making the policy more difficult to learn. In response to the above issues, this work proposes an offline RL two-stage optimization framework. The framework consists of an offline RL method based on unknown behavior to alleviate the problem of action overestimation and a two-stage feature optimization training method to reduce the impact of large spaces. Experimental results prove that the method proposed in this article can effectively improve the long-term recommendation ability of the model and more effectively improve the model performance than the general offline RL algorithm.