Improving Performance of a QA System in a Closed Domain by Question Validation

Traditional question-answering (QA) systems attempt to provide answers to all user queries, even when those queries are vague, irrelevant, or fall outside the domain’s scope. This often leads to fabricated or misleading answers, particularly in specialized or low-resource contexts. To improve QA performance in low-resource closed domain environments, we propose a question-validation layer that pre-screens input questions before answer retrieval begins. Using a QA system developed for a user interface of a SaaS platform as a case study, we implemented a semantic filter that screens queries formulated in human language based on sentence-transformer embeddings and cosine similarity. A similarity threshold balancing precision and recall metrics of performance of the validation module was empirically identified as satisficing. It supports filtering out vague or out-of-scope questions while retaining 98% of relevant queries. We also incorporate domain meta-data as embeddings to improve recall for queries that use varied phrasing. The method is applicable to other specialized domains where reliable QA is challenging due to limited labeled data.