Snap and Translate Using Windows Phone

We have developed a prototype of a mobile app called "Snap and Translate" on "Windows Phone 7". A person who is reading an English menu/sign and wants a Chinese translation of an English word or phrase or paragraph can use a Windows Phone to snap an image of the text, tap the word or swipe the phrase or circle the paragraph with a finger, and get a Chinese translation displayed on the screen of the phone. This is enabled by seamless integration of three Microsoft technologies: intelligent text extraction, OCR, and machine translation based on a client-plus-cloud architecture. The current prototype also supports Chinese OCR plus Chinese-to-English translation. In this paper, we highlight the UI design of the system and the corresponding user-intention guided text extraction approach to achieving a compelling user experience.