A CLIP-based Video Question Answering framework with Explainable AI

Video analytics utilizing Video Question Answering (VideoQA) techniques to develop AI systems capable of comprehending and responding to user-specific questions from the content of the custom video datasets. The proposed system detects objects from the custom video datasets using YOLO v8, a deep learning-based object detection algorithm driven by the provided questions. The SORT algorithm helps to track detected objects throughout the entire video. Also, significant frames are extracted from the input questions utilizing a pre-trained model Contrastive Language-Image Pretraining (CLIP) specialized for analyzing video clips. The proposed model comprises video processing where the raw video data is analyzed and transformed for further analysis. Subsequently, questions are encoded and processed facilitating the integration of natural language understanding into the system. The proposed model integrates video comprehension with questions and enables it to correlate the content of the videos with the queries posed by the user. Finally, the model generates answers to the questions asked, accompanied by the selection of the most pertinent frame from the video based on similarity scores. The explainable LIME algorithm enhances the interpretability of the frame with the highest similarity score. This selection process ensures that the provided answers are grounded in the relevant visual context, enhancing the overall quality and relevance of the responses generated by the system. The model exhibited superior performance when applied to specialized custom video datasets.