Enabling access through real-time sign language communication over cell phones

The primary challenge to enabling real-time two-way video conferencing on a cell phone is overcoming the limited bandwidth, computation and power. The goal of the MobileASL project is to enable access for people who use American Sign Language (ASL) to an off-the-shelf mobile phone through the implementation of real-time mobile video communication. The enhancement of processor, bandwidth, and power efficiency is investigated through SIMD optimization; region-of-interest encoding based on skin detection; video resolution selection (used to determine the best trade off between frame rate and spatial resolution); and variable frame rates based on activity recognition. Our prototype system is able to compress, transmit, and decode 12-15 frames per second in real-time and produce intelligible ASL at 30 kbps. Furthermore, we can achieve up to 23 extra minutes of talk time, or a 8% gain over the battery life of the phone, through our frame dropping technique.