Large-Scale Storage Location Assignment via Hierarchical Reinforcement Learning: A Rank and Assign Approach

The storage location assignment problem for export containers (EC-SLAP) is crucial to the efficiency of cargo turnover in ports. Existing methods fall short in real-world applications due to the challenges of the unpredictability of container arrival sequences and the large-scale problem. We propose a new buffer-wise framework based on hierarchical reinforcement learning for EC-SLAP, aimed at optimizing container turnover efficiency. The framework comprises two processes: 1) Ranking Agent ranks containers in the buffer, reducing the uncertainty of random arrival sequences compared to the immediate assignment. 2) Assigning Agents assign storage locations in a two-step process by block and slot, diminishing the dimensionality of the large-scale discrete action space. We iteratively optimize agents by asynchronously obtaining rewards from the environment. In addition, to address the challenge of sparse rewards in long-sequence decision-making, we have developed a novel immediate reward function to enhance learning efficiency and accelerate convergence. We propose a new large-scale dataset, NZP-SLAD, collected from real-world historical data from the terminal operating system of Ningbo-Zhoushan Port and develop a realistic container terminal simulator. We conducted numerous offline simulations and tests with this dataset. The experimental results demonstrate that our proposed method achieves rapid convergence and significantly surpasses expert methods used in real-world production.