Neural Constrained Combinatorial Bandits

Constrained combinatorial contextual bandits have emerged as trending tools in intelligent systems and networks to model reward and cost signals under combinatorial decision-making. On one hand, both signals are complex functions of the context, e.g., in federated learning, training loss (negative reward) and energy consumption (cost) are nonlinear functions of edge devices’ system conditions (context). On the other hand, there are cumulative constraints on costs, e.g., the accumulated energy consumption should be budgeted by energy resources. Besides, real-time systems often require such constraints to be guaranteed anytime or in each round, e.g., ensuring anytime fairness for task assignment to maintain the credibility of crowdsourcing platforms for workers. This bandit setting presents significant challenges, including modeling complex rewards/costs, satisfying anytime cumulative constraints, and balancing exploration and exploitation. Therefore, we propose a primal-dual algorithm (Neural-PD) with neural network-based estimations for rewards/costs and virtual queue-based optimization for constraints. Besides, we provide theoretical guarantees regarding the behavior of neural network training within the primal-dual framework and the dynamic neural tangent kernel (NTK) of the neural networks during online learning. By integrating NTK theory and Lyapunov-drift techniques, we prove Neural-PD achieves a sharp regret bound and a zero constraint violation. We also show Neural-PD outperforms existing algorithms with extensive experiments on both synthetic and real-world datasets.