A Hitting Time Analysis for Stochastic Time-Varying Functions With Applications to Adversarial Attacks on Computation of Markov Decision Processes

Stochastic time-varying optimization is an integral part of learning in which the shape of the function changes over time in a nondeterministic manner. This article considers multiple models of stochastic time variation and analyzes the corresponding notion of hitting time for each model, i.e., the period after which optimizing the stochastic time-varying function reveals informative statistics on the optimization of the target function. The studied models of time variation are motivated by adversarial attacks on the computation of value iteration in Markov decision processes. In this application, the hitting time quantifies the extent that the computation is robust to adversarial disturbances. We develop upper bounds on the hitting time by analyzing the contraction–expansion transformation appearing in the time-variation models. We prove that the hitting time of the value function in the value iteration with a probabilistic contraction–expansion transformation is logarithmic in terms of the inverse of a desired precision. In addition, the hitting time is analyzed for optimization of unknown continuous or discrete time-varying functions whose noisy evaluations are revealed over time. The upper bound for a continuous function is super-quadratic (but subcubic) in terms of the inverse of a desired precision and the upper bound for a discrete function is logarithmic in terms of the cardinality of the function domain. Improved bounds for convex functions are obtained and we show that such functions are learned faster than nonconvex functions. Finally, we study a time-varying linear model with additive noise, where hitting time is bounded with the notion of shape dominance.