MEDIFICS: Model Calling Enhanced VLM for Medical VQA

The integration of multimodal data has revolutionized medical diagnostics by combining visual and textual information for a comprehensive understanding of patient conditions. This paper introduces MEDIFICS, an innovative solution that leverages the IDEFICS-9b-instruct model, fine-tuned with QLoRA, to transform image and text data into actionable medical insights. Synthetic data generation enhances model training by creating structured datasets from multiple sources, thereby improving accuracy and generalization. The system ensures precise diagnostics through specialized models for image classification and object detection, tailored to medical specialties like dermatology, radiology, and oncology, using a new functionality called model calling. Stringent data security measures protect sensitive information, and evaluation metrics show significant accuracy improvements post-fine-tuning. Practical applications in diagnosing skin lesions, interpreting radiographs, and identifying breast masses demonstrate MEDIFICS’ potential to revolutionize medical diagnostics, offering time-efficient, accessible, and free consultations. This study highlights the effectiveness of multimodal data analysis and synthetic data generation, setting the stage for future advancements in AI-driven healthcare solutions.