A Lip-Reading Recognition Method that Integrates a 3D Dual-Stream Convolutional Neural Network

Lip reading refers to interpreting a speaker's content by observing lip movements and is widely applied in speech reconstruction, digital human modeling, and multimodal speech systems. A key challenge in this area lies in efficiently extracting temporal information. This study introduces a Temporal Residual Network to capture critical lip-reading features and proposes, for the first time, a method that uses temporal residuals to merge slow and fast dual-stream channels. Specifically, the slow-stream channel in the front-end network replaces the average pooling layer in the ResNet-18 backbone with a 3D convolution, while the fast-stream channel retains the pooling layer and incorporates channel-wise shifting via TSM. Two CNN streams with different channel capacities are utilized to extract static features from keyframes and dynamic information from non-keyframe sequences. Experimental results confirm the effectiveness of this approach. The model achieves 89.14% accuracy on the English LRW dataset, surpassing the baseline, and 58.45% on the Chinese LRW-1000 dataset, establishing a new benchmark in lip-reading technology.