Integrating Visual Data in Conversational AI: A Study on LangChain Custom Tools for Image-Based Interaction

This research investigates the development of a visual question answering (VQA) system embodied as a chatbot. The system leverages Langchain, a framework for integrating natural language processing (NLP) and computer vision (CV) tasks. Users interact with the VQA system via a chat interface built with Streamlit, a Python library for web application development. The chatbot utilizes Langchain to extract visual features from user-provided images and interprets them as part of the overall query. To enhance information retrieval, the system integrates functionalities for web search and advanced querying capabilities through Structured Query Language (SQL). Furthermore, the VQA system incorporates a memory component, leveraging past interactions to personalize future responses and improve user experience.