TedNet: A Cross-modal Alignment Network Based on Text Dependency Knowledge for Automatic Generation of Radiology Reports

Medical imaging is used in clinical practice for diagnosis and treatment, but writing imaging reports is timeconsuming and error-prone, especially for inexperienced radiologists. To reduce radiologists' workload and minimize misdiagnosis, methods for automatically generating radiology reports have gradually gained attention. However, there are two major problems with this method: (1) data imbalance: the ratio of abnormal regions to normal regions is extremely low, while the report needs to accurately describe the abnormal regions; (2) the mapping relationship between image and texts is difficult to align effectively and accurately due to the lack of annotations. Therefore, this article proposes a cross-modal alignment network based on text-dependent information to automatically generate radiology reports. In particular, this article calculates collaborative attention(co-attention) of visual features and semantic features, i.e., text dependency, to locate the abnormal region and generates a response to align the image and the text through the cross-modal alignment network. Finally, this article proposes a gated attention algorithm to incorporate text-dependent information into the decoder's multi-head attention. In this paper, extensive experiments are conducted on two benchmark datasets, IU-Xray and MIMIC-CXR. The results show an improvement of 5.6 % and $\mathbf{2. 5 \%}$ on the BLEU-3 score, achieving state-of-the-art results.