Image Synthesis of Hepatobiliary Phase using Contrast-Enhanced MRI and Diffusion Model

Hepatobiliary phase (HBP) imaging is of great value in the diagnosis of liver cancer. However, the imaging process of HBP is very time-consuming, and it needs to wait 20 minutes after the injection of contrast agent. In order to obtain HBP images more conveniently and efficiently in clinic, we propose a novel method based on multimodal image synthesis with deep learning, which generates HBP images through contrast-enhanced images and diffusion models. We first independently train the late diffusion models for generating HBP images for each modality, and pre-align all encoders through contrastive learning. Then, these diffusion models effectively learn to focus on cross-modal joint multimodal generation through a transformer mechanism called "latent spatiotemporal alignment" (LSAT). In order to model spatiotemporal characteristics of contrast-enhanced MR while maintaining visual generation quality, we propose a spatiotemporal module to construct diffusers and achieve cross-attention in the diffusion stream during the joint generation process. The results of clinical experiments demonstrate the effectiveness of the proposed method, indicating that the proposed method is superior to the typical image synthesis methods. In addition, the ablation study also demonstrates the effectiveness of the proposed modules. Finally, we quantitatively reveal the contribution of different modalities of contrast-enhanced MR to the generation of HBP images, making it easier for clinical interpretability.