Real-time, fully automatic upper facial feature tracking

Robust, real-time, fully automatic tracking of facial features is required for many computer vision and graphics applications. In this paper, we describe a fully automatic system that tracks eyes and eyebrows in real time. The pupils are tracked using the red-eye effect by an infrared-sensitive camera equipped with infrared LEDs. Templates are used to parameterize the facial features. For each new frame, the pupil coordinates are used to extract cropped images of eyes and eyebrows. The template parameters are recovered by PCA (principal component analysis) on these extracted images using a PCA basis, which was constructed during the training phase with some example images. The system runs at 30 frames per second and requires no manual initialization or calibration. The system is shown to work well on sequences with considerable head motions and occlusions.