Digital Twin-Driven Reinforcement Learning Method for Marine Equipment Vehicles Scheduling Problem

In the traditional marine equipment construction process, the material transportation vehicle scheduling method dominated by manual experience has shown great limitations, which is inefficient, costly, wasteful of human resources, and unable to cope with complex and changing scheduling scenarios. The existing scheduling system cannot realize the information interaction and collaborative integration between the physical world and the virtual world, while the digital twin (DT) technology can effectively solve the problem of real-time information interaction and the reinforcement learning (RL) method can cope with dynamic scenarios. Therefore, this paper proposed a DT-driven RL method to solve the marine equipment vehicle scheduling problem. Given the dynamic nature of transportation tasks, the diversity of transported goods, and the optimization characteristics of transportation requirements, a framework for scheduling transportation vehicle operations based on DT is constructed, and a RL-based vehicle scheduling method in a dynamic task environment is proposed. A Markov decision process (MDP) model of the vehicle scheduling process is established to realize one-to-one mapping between information and physical elements. An improved RL method based on Q-learning is proposed to solve the MDP model, and the value function approximation and convergence enhancement methods are applied to optimize the solving process. Finally, a case study is used for example verification to prove the superiority and effectiveness of the proposed method in this paper.
Note to Practitioners
—The motivation of this paper is to optimize material transportation vehicle scheduling in dynamic task environments and to improve logistics transportation efficiency. Therefore, a DT-based vehicle scheduling method for marine equipment is proposed. Firstly, a framework of vehicle scheduling based on DT is designed to establish a MDP model of the vehicle scheduling process, and the dynamic task characteristics are described by mathematical methods in the design of the elements of the model. A RL-based vehicle scheduling method is proposed. The value function approximation method and the convergence enhancement method of the algorithm are investigated for the characteristics of continuous dynamic action features leading to huge state space and non-convergence of the algorithm. The algorithm performance is verified and analyzed through data validation of actual cases.