Language-driven multimodal model for intravascular ultrasound image segmentation

IntraVascular UltraSound (IVUS) is a technique for examining coronary artery disease, frequently utilized to determine plaque burden in the assessment of coronary atherosclerosis. Calculating the plaque burden requires segmenting the lumen and vessel. Due to the nature of ultrasound, an anechoic zone will emerge behind the calcification, obscuring the entire vessel, which makes the segmentation more difficult. Doctors can reason about vascular contours based on prior knowledge that vessels are circular, if the machine can interpret this knowledge, it will aid us in solving the problem. Based on this purpose, we combine IVUS images with descriptions of IVUS intrinsic characteristics to deduce the vascular contour in the anechoic zone, using multi-scale network and cross attention module. 34 IVUS pullbacks from three 40 MHz catheters were obtained, with 720 frames in the training dataset and 240 frames in the test dataset. Simultaneously, the lumen, vessel, and calcification were segmented, and the network's performance was assessed using the Jaccard Measure (JM) and Hausdorff Distance (HD). The vessel, lumen, and calcification Jaccard measure values and standard deviations in our network were 0.93 (0.03), 0.86 (0.04), and 0.63 (0.10), respectively. Experiments demonstrated that label descriptions can increase IVUS image segmentation accuracy, particularly when reasoning about the vascular contour in the echoless zone behind the calcification.