Multimodal music retrieval for large databases

We present a novel multi-modal access to large MP3 music databases. Retrieval can be fulfilled either in a content-based manner or by keywords. As input modalities, speech by natural language utterances or singing, and manual interaction by handwriting, typing or hardkeys are used. In order to achieve especially robust retrieval results and automatically suggest music to the user, contextual knowledge of the time, date, season, user emotion, and listening habits is integrated in the retrieval process. The system communicates with the user by speech or visual reactions. The concepts shown are especially designed for home and mobile access on tablet-PCs, PDAs, and similar PC solutions, The paper discusses the concept and a working prototype called Shangrila. An evaluation by a user study leads to an impression of the capabilities of the suggested approach to multimodal music retrieval.