2D-3D Feature Co-Embedding Network with Sparse Annotation for 3D Medical Image Segmentation

Supervised methods on 3D medical image segmentation need large amounts of annotated data, but annotating is time-consuming. Also, existing 3D segmentation methods capture more global structural information but overlook local detailed features, which negatively impacts the segmentation of small tissues. In this paper, we propose a novel weakly-supervised 2D-3D Feature Co-Embedding Network (2D-3D CoENet) that includes 2D and 3D encoding layers, simultaneously extracting 2D local detailed and 3D global structural features. To reduce annotation costs, we use fewer labeled slices as ground truth and pseudo-labels are generated by 2D-3D CoENet for other slices. Additionally, multi-view learning is introduced to capture more 2D local detailed information, and a Multi-view Semantic Consistency loss (MSC loss) is proposed to constrain features from multiple perspectives. To further enhance the local detailed texture features, we propose an Edge Enhancement Module (EEM) in the 3D segmentation network to enhance the edge detail features. Our experimental results on the SKI10 dataset and OAI ZIB dataset demonstrate that our method outperforms the SOTA weakly-supervised segmentation methods. Moreover, our approach achieves results that are comparable to the fully-supervised upper bound results.