Extracting Causal Relational Rules for Medical Question-Answering Tasks using Large Language Models

Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding, summarizing, and extracting various topics from free-form text. However, most domain-specific extraction using LLMs still relies on an annotated corpus that is often expensive to achieve. We propose a prompting-based framework to extract causal relational rules from medical question-answering text without annotation. Our framework also integrates a self-judging workflow to enhance the quality of extracted rules. We also present a pilot study that evaluates the quality of the extracted rules based on human labels. The results demonstrate the efficacy of our framework in extracting high-quality rules that can enhance the performance of downstream LLMs-driven QA tasks. We have made the framework and dataset with enhanced rules available as open-source to encourage a wide range of applications.