Human-Machine Interaction Through Visual Speech Recognition

Visual Speech, colloquially known as lipreading refers to the portion of speech that is communicated and interpreted through the variations in mouth shape and facial expressions that occur when an individual speaks. It is a potential general communication methodology that presents a multitude of advantages over audio-based communication. However, it currently lacks the technological structure to support real world usability. Past research focused on applying Deep Learning techniques such as Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM) and Transformers, or Natural language Processing models such as Hidden Markov Models (HMM). However, there is little focus on enhancing performance specific to parameters such as word substitutions and deletions that are pivotal for Human-Machine Interaction (HMI). This system applies the Swish activation function to the 3DCNN architecture to address the adversary of word substitutions and deletions in the predicted visual speech that can be countered by smoother feature extractions via the swish function and lower the Word Error Rate (WER) recorded.