Experience Replay-Driven Evolutionary Reinforcement Learning for Dynamic Flexible Flow Shop Scheduling

The Dynamic Flexible Flow Shop Scheduling Problem (DFFSP) represents a critical frontier in smart manufacturing, demanding online decision-making mechanisms that optimize both production efficiency and responsiveness. However, the NP-hard and dynamic properties of DFFSP poses a significant challenge to traditional scheduling methods. This paper proposes Experience Replay-Driven Evolutionary Reinforcement Learning (ER-ERL), which integrates experience replay buffers with adaptive evolutionary operators. First, a novel Markov Decision Process (MDP) model is proposed for DFFSP, incorporating a state space with static and dynamic features, a multidimensional action space that combines job sequencing and machine allocation, and a reward function balancing job timeliness and machine utilization. Second, an experience replaydriven indicator is proposed, which measures population quality through evaluating agent behaviors. This indicator alleviates the risk of premature convergence by simultaneously assessing convergence degree and behavioral novelty. Furthermore, enhanced evolutionary operators are introduced, which dynamically balance exploitation and exploration, enhancing global search capabilities. Finally, experimental results demonstrate the superiority of ER-ERL across four workload dynamics scenarios. Specifically, ER-ERL consistently achieved at least an $11 \%$ improvement in on-time delivery rate compared to state-of-the-art scheduling algorithms.