A Smart Camera for Multimodal Human Computer Interaction

A smart camera is an embedded vision system which, in addition to image capture, performs image analysis and pattern recognition to provide as output a high-level understanding of the imaged scene. Smart cameras are essential components to build active and automated control systems for many applications, such as surveillance, machine vision, and interactive visualization systems. The heart of smart camera is the intelligent image processing algorithms that turn raw data into knowledge. The design of smart camera is challenging because on one hand video processing has insatiable demand for performance and power, and on the other hand embedded systems place considerable constraints on the design. In this paper we firstly present an overview of smart camera technologies and the process to design smart cameras as embedded systems. We then present the design and implementation of a smart camera, called GestureCam, which can recognize simple hand and head gestures. The camera uses a CMOS image sensor as capture front-end, and the image processing and gesture recognition is completely built on a single FPGA device. The experimental results have shown it to be robust with enough performance to meet real-time constraints. We plan to use the GestureCam to build next generation of natural multimodal human computer interfaces.