FrictionSegNet: Simultaneous Semantic Segmentation and Friction Estimation Using Hierarchical Latent Variable Models

This paper presents an end-to-end approach, named FrictionSegNet, for jointly estimating tyre-road friction coefficient and identifying road surfaces in real time from on board camera data. FrictionSegNet combines semantic segmentation and friction estimation by learning a shared latent space that encompasses both semantic segmentation and friction coefficient information. An objective function is designed for this task and minimised using *geco to train the model, providing the ability to control the balance between improved predictions and uncertainty measurement. To the best of our knowledge, this study is the first attempt to jointly estimate tyre-road friction and surface type by learning the joint latent space of semantic segmentation and friction coefficient information. The results suggest that it is possible to identify low-friction surfaces, e.g. snow or ice, and estimate upcoming road friction in real time from a camera only. As it is of interest to develop techniques that require less training data, numerical experiments were performed using transfer learning from a dataset consisting of images of various road surfaces. This led to better performance and faster convergence during training. FrictionSegNet achieved per-pixel accuracies of 97% and 95% when identifying snow and ice respectively, and RMS errors of 0.04–0.09 when estimating $\mu $ values achievable by a truck *abs on gravel, dry and wet asphalt, snow, and ice surfaces.