Research on a Lip Reading Recognition Model Based on Multi-Scale Texture Motion Features and Cross-Spatio-Temporal Pyramid Pooling

Lip reading recognition is a crucial technology that decodes spoken content by analyzing the movements of a speaker's lips in noisy environments, and it can also aid communication for individuals with speech impairments. However, effective lip reading requires processing information from continuous video frames, necessitating the consideration of correlations between adjacent images as well as those across longer distances. Our research emphasizes the importance of model design, particularly the adoption of a lip reading recognition model utilizing cross-spatio-temporal pyramid pooling. This novel model offers an unprecedented approach to lip reading recognition, significantly improving both performance and robustness. We first designed a multi-scale feature extraction method to capture subtle variations in lip movement, further enhancing feature representation capabilities through the application of cross-spatio-temporal pyramid pooling strategies. We used the lip-reading baseline model on the LRW and LRW1000 data and obtained accuracies of 89.5% and 56.9%. This result represents an improvement compared to previous works.