Integrating image and text information for biomedical information retrieval

Summary form only given. The search for relevant and actionable information is key to achieving clinical and research goals in biomedicine. Biomedical information exists in different forms: as text and illustrations in journal articles and other documents, in "images" stored in databases, and as patients' cases in electronic health records. In the context of this work an "image" includes not only biomedical images, but also illustrations, charts, graphs, and other visual material appearing in biomedical journals, electronic health records, and other relevant databases. The tutorial will cover methods and techniques to retrieve information from these entities, by moving beyond conventional text-based searching to combining both text and visual features in search queries. The approaches to meeting these objectives use a combination of techniques and tools from the fields of Information Retrieval (IK), Content-Based Image Retrieval (CBIR), and Natural Language Processing (NLP). The tutorial will discuss steps to improve the retrieval of biomedical literature by targeting the text describing the visual content in articles (figures, including illustrations and images), a rich source of information not typically exploited by conventional bibliographic or full-text databases. Taking this a step further we will explore challenges in finding information relevant to a patient's case from the literature and then link it to the patient's health record. The case is first represented in structured form using both text and image features, and then literature and EHR databases can be searched for similar cases. Further, we will discuss steps to automatically find semantically similar images in image databases, which is an important step in differential diagnosis. Automatic image annotation and retrieval steps will be described that use image features and a combination of image and text features. We explore steps toward generating a "visual ontology", i.e., concepts assigned to image patches. Elements from the visual ontology are called "visual keywords" and are used to find images with similar concepts. The tutorial will demonstrate some of these techniques by demonstrating our Image and Text Search Engine (ITSE), a hybrid system combining NLM's Essie text search engine with CEB's image similarity engine.