Conditional Class-Specific Prompt Learning for Medical Image Recognition

Deep learning has achieved remarkable results in medical image recognition. However, due to the scarcity of medical images, transfer learning is particularly critical. Visuallanguage pre-trained models such as CLIP have extensive prior knowledge, and prompt learning for CLIP has recently made great strides on downstream tasks involving natural images. That provides new possibilities for medical image recognition. However, the effectiveness of prompt learning is still limited for medical images that are beyond the internal knowledge of the pre-trained model. The high specialization of medical terms hinders the model’s understanding, and relying solely on text descriptions to distinguish medical images has certain limitations. To address this problem, we propose a conditional class-specific prompt (CCSP) learning method. Specifically, we first build a class-specific external memory bank which store the detailed features in medical images. Then, the features in the memory bank are converted into prompts that align with the text descriptions by category, thereby better aligning the semantic information and context of the images. During training, the text prompts are continuously updated to achieve optimal alignment. Our experiments verify the effectiveness of CCSP.