Artificially Generated Visual Scanpath Improves Multilabel Thoracic Disease Classification in Chest X-Ray Images

Expert radiologists visually scan Chest X-Ray (CXR) images, sequentially fixating on anatomical structures to perform disease diagnosis. An automatic multilabel classifier of diseases in CXR images can benefit by incorporating aspects of the radiologists’ approach. Recorded visual scanpaths of radiologists on CXR images can be used for the said purpose. But, such scanpaths are not available for most CXR images, which creates a gap even for modern deep learning-based classifiers. This article proposes to mitigate this gap by generating effective artificial visual scanpaths using a visual scanpath prediction model for CXR images. Further, a multiclass multilabel classifier framework is proposed that uses a generated scanpath and visual image features to classify diseases in CXR images. While the scanpath predictor is based on a recurrent neural network, the multilabel classifier involves a novel iterative sequential model (ISM) with an attention module. We show that our scanpath predictor generates human-like visual scanpaths. We also demonstrate that the use of artificial visual scanpaths improves multiclass multilabel disease classification results on CXR images. The above observations are made from experiments involving around 0.2 million CXR images from two widely used datasets considering the multilabel classification of 14 pathological findings. Code link: (https://github.com/ashishverma03/SDC).