UniDCP: Unifying Multiple Medical Vision-Language Tasks Via Dynamic Cross-Modal Learnable Prompts

Medical vision-language pre-training (Med-VLP) models have recently accelerated the fast-growing medical diagnostics application. However, most Med-VLP models learn task-specific representations independently from scratch, thereby leading to great inflexibility when they work across multiple fine-tuning tasks. In this work, we propose
UniDCP
, a
Uni
fied medical vision-language model with
D
ynamic
C
ross-modal learnable
P
rompts, which can be plastically applied to multiple medical vision-language tasks within a unified model. Specifically, we explicitly construct a unified framework to harmonize diverse inputs from multiple pre-training tasks by leveraging cross-modal prompts for unification, which accordingly can accommodate heterogeneous medical fine-tuning tasks within a same model. Furthermore, we conceive a dynamic cross-modal prompt optimizing strategy that optimizes the prompts within the shareable space for implicitly processing the shareable clinic knowledge. UniDCP is the first Med-VLP model capable of performing all 8 medical uni-modal and cross-modal tasks over 14 corresponding datasets, consistently yielding superior results over diverse state-of-the-art methods.