EMPATHIC: Emulating Human-Like Multimodal Personality Architecture Through Thoughtful Human-AI Conversation

In the evolving landscape of human-computer interaction, this paper introduces an innovative framework poised to revolutionize chatbot systems. Our framework, meticulously designed for emotionally aware multimodal chatbots, seamlessly integrates real-time face and emotion recognition with natural language processing. Currently in the developmental phase, the framework exhibits a distinctive proficiency in executing a diverse array of human instructions. These instructions span tasks such as generating detailed captions, object counting, and responding to general queries. Leveraging parameter-efficient fine-tuning from OpenFlamingo, augmented by the inclusion of the Low-rank Adapter (LoRA), our framework strategically prioritizes versatility and operational efficiency. The core of our approach lies in the establishment of instruction templates that intricately merge both vision and language data. This strategic integration forms the backbone of our multi-modality instruction tuning methodology, enhancing the adaptability of the chatbot. Recognizing the pivotal role of high-quality training data, we emphasize its significance in shaping the dialogue performance of the chatbot. As the framework progresses through its developmental stages, this research lays a robust foundation for the creation of a versatile conversational agent. The envisioned applications of this agent range from enhancing customer service experiences to providing valuable support in mental health scenarios.