Multi-Label Concept Classification in Imaging Entities of Biomedical Literature Using CNN and Vision Transformers

Biomedical images are frequently used in articles to illustrate medical concepts and highlight regions-of-interests (ROIs) by using annotation markers (pointers) such as different arrows, letters or symbols overlaid on figures. Also, in many cases multiple markers in the same image are often pointing to different concepts relevant to the article. Hence, each image might be assigned with one or more concepts for multi-label classification and object detection based machine-learning tasks. This work reports such a proof-of-concept (POC) experiment by annotating ROIs and classifying (multi-label classification) 200 Chest CT images appeared in biomedical articles with eleven (11) different concept (similar to UMLS) categories such as, ground-glass, bronchi, honeycomb, cyst, nodules, etc. For annotation, we use an online tool (Labelimg) to annotate image ROIs with concepts based on the information content in associated captions. To demonstrate the feasibility of the POC, this study conducts experiments with different Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) using both transfer learning (fine-tuning) and training from scratch. We achieved encouraging results (around 70% micro average precision and recall accuracies) in a test set, whereas the dataset images are in very low resolution, non-uniform lighting conditions and with varied shapes and sizes. Overall, this study demonstrates the effectiveness of deep learning models in multi-label classification in medical images and establishes the feasibility and rationale of the POC. The ultimate goal of this work is to develop a large-scale concept detection framework towards building a visual ontology of images in biomedical articles.