Building Lightweight Domain-Specific Consultation Systems via Inter-External Knowledge Fusion Contrastive Learning

Large language models (LLMs) have demonstrated their vast potential and value in natural language processing tasks and beyond. However, when these models are applied to develop consultation systems for industrial domains, such as e-government, intelligent diagnosis, and legal consultancy, they encounter many unresolved technical issues. These include the vast scale of the models, lack of tight fusion with existing industry knowledge, along with occurrences of model hallucinations and inadequate explainability. Unlike general-purpose dialogue systems, building a consultation system for a specific industrial domain requires not only the integration of extensive external knowledge from the Internet but also the incorporation of precise, specialized knowledge from specific industries, making the challenges even more complex. In response to these challenges, we propose the Inter-External Knowledge Fusion Contrastive Learning Technique. This technique facilitates the integration of internal industry knowledge with widely accessible external knowledge from the Internet and provides a universal framework for building lightweight, domain-specific consultation systems. Utilizing this technique enables the straightforward creation of precise, professional, and constantly updated domain-specific consultation systems applicable across various industries. Overcoming the inherent limitations associated with LLMs, this technique achieves a performance level comparable to LLMs. To validate the effectiveness of our proposed technique, we conducted extensive experiments in the development of real-world industry consulting systems. By testing on seven real datasets covering diverse tasks, we demonstrate the system’s exceptional performance: our lightweight consultation system utilizes only 4% of the parameters of an LLM but achieves over 90% of its performance level.