RAG-Enhanced Large Language Model for Intelligent Assistance from Web-Scraped Data

The explosion of online information necessitates efficient and accurate methods for retrieving and analyzing relevant data. This research proposes a novel framework that leverages Retrieval-Augmented Generation (RAG) techniques to address this challenge. The framework utilizes web scraping to extract information from various sources and employs advanced language models like Llama 3, Mistral, and Gemini to generate concise and informative summaries. Furthermore, the framework incorporates embedding models such as nomic-embed-text and Gemini embedding model text embedding 001 to create semantic representations of the retrieved data and utilize FAISS for efficient indexing and retrieval. The RAGAS framework was used to evaluate the performance of different LLMs, with Llama 3.1 demonstrating the highest accuracy at 86.67%. This framework has significant implications for various applications, including personalized education, where it can be used to provide students with tailored learning materials and personalized tutoring experiences.