Multimodal Representation Learning for Blastocyst Assessment

Blastocyst selection based on morphology grading is crucial in in vitro fertilization (IVF) treatment. Several research studies based on convolutional neural networks (CNNs) have been reported to select the most viable blastocyst automatically. In this paper, we propose a multimodal representation learning framework in which the text description is firstly streamed as a complementary supervision signal to enrich the visual information. Moreover, we redefine the blastocyst assessment problem to an image-text retrieval task to solve the data imbalance. The experimental results show that the performance metrics, e.g., accuracy, outperform the unimodal classification (+1.5%) and image retrieval counterparts (+1.2%), which demonstrates our proposed model’s effectiveness.