Multimedia database retrieval using hand-drawn sketches

We propose an integrated approach to color and shape based retrieval which enables the user to search a color image database intuitively by presenting simple sketches. Each of the images is represented by a Hidden Markov Model (HMM) which has been concatenated with modified filler models in order to obtain scale and rotation invariance properties. These properties are particularly important when using query by sketch due to the skew which occurs naturally in human handwriting and in drawings. The use of streams (sets of features that are assumed to be statistically independent) within the HMM framework allows the integration of shape and color derived features into a single model, thereby allowing one to control the influence of the different streams via stream weights. The approach has been evaluated on a color image database containing 120 different isolated objects with arbitrary orientation and showed good retrieval results with several users. Furthermore, the use of HMMs allows efficient pruning and thus a fast retrieval even with large databases.