AzureIQ-RAG: Hybrid RAG System for Automated Information Retrieval and Semantic Query Answering Using Azure AI

Extracting relevant information from documents and websites is a time-consuming task that becomes increasingly complex as document length grows and websites expand. Manually retrieving precise information to meet user requirements from large documents and extensive websites is challenging and inefficient. In this context, Artificial Intelligence (AI) has emerged as a transformative technology, offering unparalleled capabilities for automating systems. AI has been widely utilized to streamline various processes, demonstrating its potential to simplify complex tasks, enhance accuracy, and reduce human effort. This research paper introduces "AzureIQ-RAG" Azure powered Intelligent Query and Hybrid Retrieval Augmented Generation (RAG) system with two primary features: (1) extracting and retrieving information from diverse document formats, including PDF, TXT, DOCX, etc. and enabling question-answering based on the provided document(s), (2) extracting content from website and enabling question-answering regarding the content of the website. The system automates the extraction process from both documents and websites. The extracted content is segmented, embedded using Azure OpenAI Embeddings, and indexed in Azure Cognitive Search for efficient retrieval and then a reranking mechanism is applied to improve precision and accuracy further. Then the final response is generated by passing the relevant content and user query to the Large Language Model (LLM) GPT -4 omni. Additionally, the system maintains separate session memory for both document and website searches, ensuring context retention for improved query handling and user experience. The document processing feature of the system has been tested on a dataset of over 950+ documents of various formats and achieved an accuracy of 94.77 %, and the website feature of the system is tested on 50+ different websites, achieving an accuracy of 75.22%.