A Comparison of Different State Representations for Reinforcement Learning Based Variable Speed Limit Control

Variable Speed Limit Control (VSLC) is one control method for alleviating congestions on urban motorways. Machine learning techniques, like Reinforcement Learning (RL), are a promising alternative for setting up VSLC because an optimal control policy can be achieved with a smaller computational burden in comparison with optimal control approaches. A drawback is a large number of learning iterations and the problem of the exponential expansion of the state space dimension. This can be solved with function approximation techniques. Three different approaches for feature-based state representation in RL based VSLC are compared in this paper regarding the convergence of Total Time Spent. The microscopic traffic simulator VISSIM with a representative traffic model is used to evaluate the compared approaches. Results show that function approximation methods outperform RL based VSLC formulated with a lookup table by an average improvement of 10 %, where feature extraction methods (Coarse and Tile) coding showed slightly faster learning rate.