AV-CPS: Audio Visual Cognitive Processing System for Critical Intervention in Autonomous Vehicles

Autonomous Vehicles (AVs) are promising solutions for safe and comfortable driving. The evolution of AVs are driven by various technical and socio-economic factors. Autonomous navigation in AVs is the ability to plan and navigate towards a destination, while dealing with the path's obstacles. Current research solutions rely on sensors and inter-vehicular communication to perceive the environment. But they are susceptible to threats ranging from physical damage to security issues. Hence, to improve the AV's perception capabilities, this article proposes a multi-modal mechanism known as Audio-Visual Cognitive Processing System (AV-CPS), which combines speech and gesture inputs from users to enable spontaneous passenger control for AVs. The results demonstrate over 20% improvement in action prediction accuracy over uni-modal mechanisms and a relative reduction in response time from the simulation of the AV-CPS.