Lip Reading using Video and Thermal Images

This paper describes the lip reading method using video and thermal images. We pay attention to the lip movement. The density of the thermal image expresses the surface temperature, and it is difficult to detect the lip position automatically. Though the video and the thermal images take in the same time, these don't synchronized in time and spatially. Then the time matching and the spatial matching process are applied. Next, a lip region is detected from the video image, and a lip region of the thermal image is detected corresponding to the video region. Finally, the eigenimage waveform is computed, and DP matching is applied. Experiments were conducted for 75 scenes (five words of three subjects). A successful recognition rate of 76.0% with the video image, 44.0% with the thermal image, and 80.0% with both images were obtained. Experimental results show the recognition rate was improves by combining with the video and the thermal images