REMDoC: Reference-Free Evaluation for Medical Document Summaries via Contrastive Learning

Despite significant advancements in automatic summary evaluation metrics based on pre-trained language models, accurately assessing the quality of medical document summaries remains a considerable challenge. Existing evaluation metrics often struggle to provide reliable assessments for medical summaries, particularly in the absence of reference texts. In this paper, we propose novel reference-free medical document summary evaluation metric, REMDoC: Reference-free Evaluation for Medical Document Summaries via Contrastive Learning which does not require reference summaries to evaluate summaries. REMDoC employs contrastive learning using medical text-tailored data augmentation techniques. Our primary motivation is to improve the alignment of automatic evaluations with human judgments, making the evaluation process more reliable and closer to medical expert assessments. Our research showcases the metric’s superior performance in assessing the quality of generated summaries without the need for comparison texts. Through extensive experimentation and analysis, this work makes significant strides in improving the reliability and usability of automatic medical document evaluation tools in medical document settings.