Progressive Goal-Oriented Communications for Reinforcement Learning Control Over Multi-Tier Computing Systems

The converging trends of reinforcement learning (RL) control and cloud-fog automation in industrial cyber-physical systems impose multiple challenges for communications to cope with stringent requirements in latency, reliability, control effectiveness and bifurcating user demands. Progressive goal-oriented (GO) communication is a promising technology to tackle the above challenges. This paper takes a two-step approach to design the first progressive codec of GO communications tailored for RL control tasks. The first step is to design a variable-rate coding scheme that extends the boundaries of rate regimes. This step is achieved by empowering the hierarchical variational autoencoder (HVAE) framework with novel algorithms such as mutual information based soft state abstraction (MISA). The second step is to transform variable-rate encoding into progressive encoding. This is achieved by applying residual-based encoding techniques upon latent representations learned by deep neural networks. Experiments on the Cartpole Swingup task demonstrate that the proposed progressive codec can facilitate smooth transitions from the ultra-low rate regime to regular rate regime, while achieving the state-of-the-art performance in terms of rate-distortion-effectiveness tradeoff.