Toward Diffusion-Based Deep Reinforcement Learning for Discrete Decision-Making: Methods and Evaluations

Discrete decision-making problems, such as combinatorial optimization tasks frequently encountered in network science and industrial IoT systems, are pervasive in real-world applications. Deep reinforcement learning (DRL) has emerged as a promising approach for addressing such challenges, offering the potential for adaptive and scalable solutions. However, existing DRL methods often suffer from limited policy expressiveness and inefficient exploration, particularly in discrete action spaces. Diffusion models, with their ability to construct rich intermediate representations through iterative denoising processes, offer a compelling alternative for enhancing discrete-action policy learning. In this work, we propose a diffusion-based online DRL approach for discrete decision-making spaces, employing denoising diffusion implicit models to accelerate policy inference. Exploration and learning efficiency are further enhanced by a hybrid memory system comprising an online replay buffer, a diffusion-specific buffer, and a high-return trajectory buffer, within which critic-guided logit construction is integrated as part of the overall training pipeline to align policy targets with both value-based and expert signals. The proposed method is evaluated on selected Gym benchmarks, classical combinatorial problems (CVRP, Knapsack, GraphColoring), and an industrial scheduling scenario. While the findings remain preliminary, the results suggest that this generative approach may offer benefits in certain contexts, particularly in improving convergence dynamics and solution robustness. The framework is intended as an initial step toward integrating generative modeling into online DRL for discrete-action spaces.