HyQA: Hybrid Near-Data Processing Platform for Embedding Based Question Answering System

An Large Language Model (LLM)-based question-answering (QA) system has gained attention for its conversational ability. However, domain knowledge limitations, time lag, high training costs, and security concerns suggest building on-premise QA systems with embedding techniques. However, deploying embedding-based QA systems on existing GPUs or domain-specific accelerators is sub-optimal as they only address high computation costs and ignore large memory footprint and data movement costs, which impact response latency and user experience. To address these issues, we propose a hybrid near-data processing platform, HyQA, which collaboratively optimizes response latency, memory footprint, and data movement cost by exploiting the benefit of near-memory and near-storage computing simultaneously. First, HyQA analyzes computational patterns of sub-tasks in embedding-based QA systems, tailors domain-specific hardware accelerators, and assigns suitable computational paradigms. Second, these dedicated accelerators are designed to communicate directly with flash memory, avoiding additional data movement. The experiment shows that HyQA significantly improves performance and reduces energy over CPU, GPU, Cognitive SSD, and DeepStore platforms.