Multimodal Integration, Fine Tuning of Large Language Model for Autism Support

This research project centres on creating an inclusive environment for the specially-abled. Created an app that integrates real-time object detection and speech-to-text capabilities which provides a multi-modal user experience with the pre-trained Model. Incorporated parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA) to emulate the behaviour of an autism counsellor.The second module of the research employs YOLOv5 and machine learning to analyze therapy videos to get interaction timestamps and statistical insights. It automates the identification of interactions and serves as a valuable aid to guardians and therapists.This paper details the methodology for real-time object detection using YOLO, SSD Mobilenet and enhances understanding in the context of adaptation of Language Models for a particular use case. Furthermore, dives deeper into the intricacies of interaction monitoring using bounding boxes in YOLO.By pushing the boundaries of Large Language Model capabilities, this research not only refines language models but also lays the foundation for their application in dynamic contexts of the real world, exemplifying their revolutionary potential.