Integrated Scheduling Optimization for Automated Container Terminal: A Reinforcement Learning-Based Approach

Container terminals face tremendous pressure to improve their throughput due to the expanding global shipping market. As a key for throughput, handling capacity requires effective coordination between various automated facilities. Observed from the operational practice of Tianjin port, a world-leading smart port, four critical facilities, namely quay cranes, lock stations, intersections, and yard cranes, are identified as bottlenecks that impact handling efficiency. Congestion at these facilities, in particular, pose significant challenges to terminal managers. To address these issues, we investigate an integrated terminal scheduling problem and formulate this novel problem as a mixed-integer linear program, from which we derive two efficient lower bounds. To tackle practical-sized problems, we propose a reinforcement learning (RL)-based algorithm with two modules. The offline module uses RL to learn from abundant historical data. When actual instance information is available, the online module enhances offline decisions using a rollout mechanism and mathematical programming. The proposed algorithm employs a pre-trained offline policy to handle extensive computations before actual decision-making and an online phase that provides a streamlined and stable method to enhance the solution. Extensive experiments validate the effectiveness of the proposed algorithm, demonstrating an 18.22% reduction in makespan compared to the rule-based heuristic used in actual port operations.