LLM-Therapist: A RAG-Based Multimodal Behavioral Therapist as Healthcare Assistant

Systems and services based on Large Language Models (LLM) are trending in the applied field of Artificial Intelligence (AI). The emergence of GPTs and subsequent improvements such as, unsupervised pre-training and transformer architecture have lead to its ability to generate human-quality text and understand context effectively. This has opened huge opportunity in healthcare domain to use AI-based assistant systems that can offer tailored recommendations and guidance for patients in numerous scenarios by considering multimodal data. This paper proposes LLM-Therapist as a multimodal personalized health care assistant for various types of patients. The proposed system uses Retrieval Augmented Generation (RAG) technique to improve the quality, accuracy, and relevance of generated response, which is specially important in providing healthcare assistance. We conducted experiments with LLM-therapist by extracting knowledge from domain-specific resources in mental health and patient’s health data. Our experiments showed better efficiency and performance in providing personalized assistance.