Multi-modal Broad Learning System for Medical Image and Text-based Classification

Automatic classification of medical images plays an essential role in computer-aided diagnosis. However, the medical images arise from the small number of available data and the improvement of existing data-enhancement methods are limited. In order to fulfil this demand, a Multi-Modal Broad Learning System (M
²
-BLS) is proposed, which has two subnetworks for simultaneous learning of both medical images and the corresponding radiology reports. M
²
-BLS provides two advantages: i) our M
²
-BLS has closed-form solution and avoids iterative training, once the image feature is available; ii) benefit from the simultaneous learning of both image and text data, our M
²
-BLS achieves high accuracy for medical classification. Experimental results on the publicly available datasets IU X-RAY and PEIR GROSS_895 show that our M
²
-BLS highly improves the classification performance, compared to SOTA deep models that learn single-type of data information only.