MT-NeT: Multi-Modality Transfer Learning Network for Automated Carotid Vessel Wall Segmentation using Sparse Annotation in MRI

Vessel Wall Magnetic Resonance Imaging (VW-MRI) is an emerging technique for visualizing lumen and vessel wall structures and facilitating the diagnosis of vascular diseases such as atherosclerosis. However, annotations on VW-MRI are usually sparse due to their labor-intensive nature. On the other hand, computed tomography angiography (CTA) images are widely used in atherosclerosis analysis, where data and annotation are relatively sufficient. To this end, we propose a multi-modality transfer learning network (MT-Net) to transfer anatomical knowledge of vessels from CTA to MR, based on fully-annotated training CTA images and sparsely-annotated training MR images. Furthermore, in the MR branch, we utilize the vessel lumen results to guide the multi-channel network for final vessel wall segmentation. Experimental results on the COSMOS Challenge dataset demonstrate advantage of our method in producing robust lumen and vessel wall segmentations with sparse annotation.