Audio-Visual Speech and Gesture Recognition using Dual Sampling Residual Attention CNN on Mobile Devices

Speech and gesture recognition has become a critical feature in this day’s applications and is critical in accessibility and learning and human-computer interfaces. However, real-scene environments raise more challenges in terms of lighting changes, head motion, acoustic environment, and background noise affecting response reality and reliability of the model. Current audio visual recognition models are not well suited to these challenges especially in the mobile environment. It is a sad reality that models trained with constrained data sets often times fail terribly when exposed to a range of conditions different from that of the training set. The current work develops a new framework for Audio-Visual Speech and Gesture Recognition (AVSGR) on the mobile platform, which employs the Dual Sampling Residual Attention Convolutional Neural Network (DSRA-CNN) enhanced through Green Anaconda Optimization (GAO) algorithm. Two publicly available datasets were utilized in this research: the AVSR AUTSL dataset that is one of the hardest because of variations in head pose, illumination and acoustic conditions or the Lip Reading in the Wild (LRW) dataset. In DSRA-CNN design, we combined two sampling rates for audio and video: one for temporal feature capture and the second one for spatial- temporal feature extraction with a reduced computational effort. In Reasoning, the method of residual attention in DSRA-CNN enables the model pay attention only to the important information to enhance recognition performance despite complicated environments. In the final analysis, GAO tune values even more optimally, increasing model efficiency and transplanting the DSRA-CNN to mobile elements. The introduced approach attains higher accuracy as 99%.