Universal multi-modal deep network for classification and segmentation of medical images

Medical image processing algorithms have traditionally focused on a specific problem or disease per modality. This approach has continued with the wide-spread adoption of deep learning in the last 5 years. Building a system with multiple neural networks and different specialized image processing algorithms is a challenge as each network requires a lot of memory and is computationally heavy. More importantly, cascading multiple networks propagates errors from one stage to another reducing overall system accuracy. In this work, we propose a single universal network that can: 1) segment different organs across different modalities, and 2) solve both segmentation and classification problems simultaneously. We compare our approach with traditional segmentation network for each modality. Our results showed modality/viewpoint classification accuracy of 99% and average dice score of 0.89 for segmentation accuracy. The proposed network can be further developed to include segmentation of more organs and disease classification.