Detecting Phonetic Characters using Radar Data

Speech recognition systems are crucial for enabling human-computer interactions, particularly for the hearing- impaired, who use them to communicate with the listening community. However, conventional speech recognition techniques faced new challenges during the Coronavirus outbreak, as protective masks obscured lip movements. To address this, a new Radio Frequency (RF) based lip-reading framework has been proposed that can read lips under face masks. This framework uses radar technology to enable RF-sensing-based lipreading, presenting the Doppler changes in the received signal due to lip movements in the form of time-frequency diagrams.To classify phonetic characters, the system uses a multi-layered deep learning neural network called MobileNet-v2, which has been optimized to achieve high accuracy on a validation dataset while avoiding overfitting the training data. The optimization process involved conducting thorough research on 12 versions of the network, each with its unique hyperparameter combination, and evaluating their performance on the validation dataset to identify the version that showed the most promising results in generalizing to new data. The selected MobileNet-v2 version achieved an impressive classification performance of 99.16% accuracy when applied to classify the considered phonetic characters.The system’s performance and testing metrics include the confusion matrix, accuracy calculation, and the ROC Curve graphic. Overall, this RF-based lip-reading framework has significant potential in enhancing speech recognition systems’ performance, particularly during situations where conventional techniques face limitations, such as mask-wearing during pandemics.