Solving Online 3D Multi-Bin Packing Problem with Deep Reinforcement Learning

The online 3D multi-bin packing problem (3D-MBPP) has wide-ranging applications, such as baggage packing, robotic arm stacking. As a classic combinatorial optimization problem, existing approaches for the 3D-MBPP primarily rely on heuristic algorithms or a combination of heuristics and machine learning. In this paper, we propose an end-to-end deep reinforcement learning framework that leverages Transformer model and Convolutional Neural Networks (CNN) to efficiently solve the 3D-MBPP. To address the searching problem of excessively large action space, the packing process is divided into three sequential sub-actions. Furthermore, a conditional querying mechanism is introduced to prevent information loss between subactions, ensuring that each sub-action preserves the complete Markov property. The framework is optimized using the Advantage Actor-Critic (A2C) reinforcement learning algorithm with Generalized Advantage Estimation (GAE). Experimental results show that the proposed method improves packing space utilization and demonstrates good generalization capabilities.