Safe Learning in Stochastic Continuum-Armed Bandit With Constraints and Its Application to Network Resource Management

This paper studies the problem of stochastic continuum-armed bandit with constraints (SCBwC), where we optimize an unknown reward function subject to an unknown constraint function over a continuous space. We model reward and constraint functions via Gaussian processes (GPs) and propose a Rectified Double-Optimistic Learning framework (RDOL), a penalty-based method incorporating double-optimistic GP bandit learning for reward and constraint functions, respectively. We consider the metric of cumulative constraint violation, which is strictly stronger than the traditional long-term constraint violation. The rectified design for the penalty update and the optimistic learning for the constraint function in RDOL guarantee the cumulative constraint violation is minimal. RDOL can achieve sublinear regret and cumulative constraint violation for SCBwC and its variants (e.g., under delayed feedback and non-stationary environment). These theoretical results match their unconstrained counterparts. We implement the framework into the problem of online resource allocation in data centers. The experimental results justify that RDOL outperforms several existing baseline algorithms.