A Novel Human-Machine Collaboration Approach for Autonomous Driving with Hand Gesture-based Guidance

In highly automated driving vehicles, a human-vehicle interface might still be required for individualization and emergency intervention. We propose a tactical human-vehicle collaboration framework by leveraging the hand-landmark extraction algorithm and the augmented reality visual feedback. The proposed vision-based interface projects the gesture, as the driver’s intention, onto the ground and feeds the projection back to the driver through the AR-HUD interface. The projected intention functions as a strategic decision or planning suggestion to the vehicle while collision avoidance, traffic rules compliance, and precise control are realized by the automation algorithm. The feasibility of the framework is validated through an integrated self-driving algorithm combining the risk field, learning-based trajectory prediction, and model predictive control. Comparisons with the conventional manual driving scheme demonstrate that high-level collaboration vastly reduces human physical burdens without compromising driving performance and driver mental workloads.