Application of Vision-Language Models for Assessing Osteoarthritis

Osteoarthritis (OA) poses a global health challenge, demanding precise diagnostic methods. Current radiographic assessments are time-consuming and prone to variability, prompting the need for automated solutions. The existing deep learning models for OA assessment are unimodal single-task systems and they don't incorporate relevant text information such as patient demographics, disease history, or physician reports. This study investigates employing Vision-Language Processing (VLP) models to predict OA severity using X-ray images and corresponding reports. Our method leverages X-ray images of the knee and diverse report templates generated from tabular OA scoring values to train a CLIP (Contrastive Language-Image Pre-Training) style VLP model. Furthermore, we incorporate additional contrasting captions to enforce the model to discriminate between positive and negative reports. Results demonstrate the efficacy of these models in learning text-image representations and their contextual relationships, showcase potential advancement in OA assessment, and establish a foundation for specialized vision-language models in medical contexts.