Learning to Sample a Signal through an Unknown System for Minimum AoI

In this work, we consider the problem of minimizing age of information (AoI) by optimizing the sampling strategy for transmitting a signal through a system with an unknown delay profile. Our goal is to observe the age at the receiver and learn online how to sample to minimize the average age. We begin our investigation with a special case of a lossless single-server system with unknown service time distribution, and decide after each packet is received how long to wait before sampling and transmitting a new packet. We derive the optimal threshold policy for an exponential server, which is able to learn the optimal policy online by estimating the service rate, and is even able to adapt when there is an abrupt change in the service rate. However, we show that this approach can fail when the distribution is not exponential, so we consider a reinforcement learning (RL) approach (Sarsa with tile coding) that does not rely on the exponential server assumption. We adapt the RL approach to the AoI minimization problem, and show that it outperforms the other estimated threshold policy in the case where the distribution is not exponential. Lastly, we discuss extensions to the problem that do not assume just a single server, and propose some approaches to solve the more general problem of sampling for an unknown system.