AR-Based Remote Command and Control Service: Self-Driving Vehicles Use Case

The recent technological advances in many fields have significantly contributed to the development of the Advanced Driver Assistance System (ADAS), which in turn will greatly contribute to the flourishing of self-driving vehicles that can operate autonomously in all road scenarios. Until then, keeping the human input in the loop remains vital to either make decisions in unseen situations or approve vehicles' proposed decisions. In this article, we leverage VR technology to provide remote assistance for self-driving in critical situations. Specifically, we study the delivery of a 360° live stream at high resolution (4K) to a remote operation center for supporting self-driving vehicles' decisions when, for example, merging onto the highway. The 360° video stream will be consumed by a human operator wearing a head-mounted display for increased flexibility, faster control, and an immersive experience. In addition, the 360° stream is augmented with relevant context data, such as the vehicle's speed and distance to other road objects, in order to increase the human operator's awareness of the vehicle and its surroundings. Depending on the human operator's proximity to the source, the video stream can either be viewed through the cloud or the edge, which further reduces the glass-to-glass latency. Experimental results demonstrate the effectiveness of employing VR technology to remotely command and control self-driving vehicles in critical situations. The results show that a 360° stream at 4K resolution can be delivered in sub-second glass-to-glass latency, which allows the operator to make timely decisions.