An End-to-End Pipeline Integrating Segmentation, Anomaly Detection, and Report Generation for Medical Imaging Analysis

Medical imaging plays a pivotal role in diagnosing and treating a variety of conditions, from brain abnormalities to retinal diseases. However, interpreting large volumes of imaging data remains an arduous and time-consuming task for radiologists, necessitating the development of automated diagnostic tools. In this paper, we present an integrated end-to-end pipeline that leverages state-of-the-art techniques in segmentation, anomaly detection, and medical report generation to improve both accuracy and efficiency. At the core of our approach lies a modified Visual Large Language Model (VLM), adapted with medical-specific adapters to transition from natural image analysis to medical image diagnostics. Our system, tested in few-shot (k=4) learning settings, demonstrates superior performance with over 90% Area Under the Curve (AUC) for anomaly detection and segmentation across diverse domains such as Brain MRI, Chest X-ray, and Retina scans. Post-anomaly detection, a custom labeling model performs multiclass classification of anomaly regions, assigning severity scores to detected abnormalities. A fine-tuned Open source Large Language Model (LLM) then generates comprehensive diagnostic reports that not only summarize findings but also suggest further diagnostic pathways. Compared to state-of-the-art models such as AprilGAN, our system demonstrates notable improvements in accuracy, adaptability, and clinical utility. These contributions establish our pipeline as a promising tool for enhancing medical imaging workflows.