Annotating Images with Drawings Through Gameplay for Drawing Based Image Retrieval

Drawing based image retrieval offers an intuitive and engaging way to retrieve images from large repositories, and can be easily performed on touchscreen devices. However, there is usually a large visual gap between a drawing query and a desired image since most users are not drawing experts. Based on the hypothesis that color drawings aiming at the same target image by different users would be more similar to each other than between a drawing and the target image itself, we propose the novel concept of image annotation with drawings for drawing based image retrieval. Just as text based image retrieval would retrieve images through retrieving their associated annotation texts, drawing based image retrieval is performed through the retrieval of the annotation drawings of the images. To address the issue of creating drawing annotations for images, we design and implement an entertaining drawing game on mobile devices based on the concept of game with a purpose (GWAP). We have so far collected over 1200 drawings of 60 target images from 30 users who play our game. We will present experimental results to demonstrate the correctness of our hypothesis. Comparing with the traditional approaches that directly comparing drawings with target images, we show that drawing based image retrieval through retrieving drawing annotations of the target images significantly improves the retrieval accuracy.