Online Deep Reinforcement Learning Approach for Solving Dynamic Vehicle Routing Problems with Time Windows

To address the limitations of existing methods for dynamic vehicle routing problems, including exponential growth in solving time as problem size increases, susceptibility to local optima, and difficulty in handling stochastic demands and changes, we propose an online deep reinforcement learning algorithm for solving dynamic vehicle routing problems with time windows. First, we establish a partially observable Markov decision process (POMDP) model based on the time sequence of customer deliveries by the fleet. This model describes the interaction between the agent and the dynamic environment. Second, we propose a network model based on the Transformer architecture with multi-head attention to represent the policy and value function networks. Additionally, an actor-critic algorithm with a greedy baseline is designed to train the neural networks. Once trained, the network model can provide end-to-end solutions for dynamic vehicle routing problems and adjust planned routes online based on dynamic changes in problem conditions. Experiments demonstrate that the proposed method outperforms advanced meta-heuristic algorithms and deep reinforcement learning algorithms in both solution speed and quality, indicating that this method can efficiently achieve online optimization for dynamic vehicle routing problems with time windows.