A visual front-end for a continuous pose-invariant lipreading system

Having an audio-visual automatic speech recognition (AVASR) system which can recognise what a speaker's says regardless of head position (i.e. left profile, front, right profile etc.), would be most useful as it enables this technology to be used in a host of realistic applications such as mobile phone and in-vehicle speech recognition. A major hurdle in achieving this goal is in developing a visual front-end which can effectively locate and track a user's face and facial features from a single camera. In this paper, we describe a visual front-end which incorporates a pose-estimator in conjunction with a parallel series of pose specific face and facial feature classifier based on a boosted cascade of simple classifiers devised by Viola and Jones [6]. Results of our visual front-end are tested on the CUAVE database. We also give lipreading results on the CUAVE database, which shows that AVASR whilst a speaker is moving their head is indeed achievable.