Analysis of a Multimodal Ocular Ultrasound Image Classification Algorithm Based on YOLO and VanillaNet

The generation of traditional ophthalmic ultrasound examination reports primarily relies on clinicians manually entering information during procedures. This process is not only inefficient and highly subjective but also heavily dependent on the clinician’s expertise. Training skilled ophthalmologists is time-consuming and costly, and with limited medical resources at the primary care level, diagnosing and treating ocular conditions faces significant challenges. To address these issues, this study proposes a multimodal AI-based method for automating report generation from ocular Doppler ultrasound images. The method integrates image features and textual information to facilitate unified disease diagnosis and report generation. The proposed approach consists of three collaboratively functioning modules:1) Multi-class Object Detection Module: Based on the YOLOv11s model, this module accurately localizes the orbital region, achieving a detection accuracy of 97.96%. 2) Text Extraction Module: By combining YOLOv11s and PaddleOCR, this module identifies key information within ultrasound images, achieving a text recognition accuracy of up to 99.9%. 3) Eye Disease Classification Module: Utilizing an improved VanillaNet architecture, this module can identify seven common ophthalmic conditions with a classification accuracy of 95.5%. This approach effectively integrates visual image features with semantic text information through a multimodal information fusion mechanism. It enables intelligent analysis and automated report generation for ocular ultrasound imaging, providing strong support in alleviating the shortage of professional ophthalmologists and holding significant potential for clinical applications.