On Resource Allocation for Remote Control of MDPs Over Queues using Age-of-Loop

We consider a wireless networked control system scenario consisting of a plant and a remote controller. The plant is modelled as a Markov decision process (MDP). The plant is connected to the remote controller through a wireless access point. At a time, the wireless access point can either transmit state observations from the plant to the controller through an uplink queue or control actions from the controller to the plant through a downlink queue. We consider a resource allocation problem for the uplink and downlink queues. The objective is to maximize control performance measured using average reward for the MDP. We also investigate whether the resource allocation problem can be equivalently solved by minimizing the Age-of-Loop (AoL), which is a recently proposed information freshness metric for closed-loop control applications. We analytically characterize the average reward in terms of the joint distribution of uplink and downlink Age-of-Information for two service disciplines and a special class of MDP models. The analytical characterization is useful for optimal uplink and downlink resource allocation. We also analytically characterize the average AoL. In contrast to prior work, we conclude that resource allocation to minimize the average AoL generally does not lead to maximization of the average reward for remote control of MDPs. We present examples of MDPs for which minimization of average AoL leads to a minimum for the average reward.