Cross-Domain Integration for General Sensor Data Synthesis: Leveraging LLMs and Domain-Specific Generative Models in Collaborative Environments

Synthetic data has emerged as a critical component in the fields of machine learning and data science, providing a solution to overcome limitations associated with real-world data, including scarcity, privacy concerns, and the high cost of acquisition. While the existing generative models demonstrate promising results in synthesizing various sensor data, they are struggling to enhance performance and generalization. In this article, we introduce a large language models (LLMs)-driven framework that leverages the power of LLMs in concert with various domain-specific generative models (DGMs) for general sensor data synthesis. Specifically, our method employs LLMs as the core to interpret task requests, decompose a complex task into a manageable set of subtasks, and delegate each subtask to the most suitable DGM, thereby automatically constructing customized data generation pipelines. Meanwhile, DGMs contribute their expertise to generate high-fidelity, domain-relevant data, whose specialized knowledge can be further enhanced by the LLM’s broad linguistic knowledge via knowledge transfer. In addition, the integration of diffusion model (DM)-based reinforcement learning (RL) is promising to enhance the framework’s ability to optimally utilize DGMs, resulting in data generation with superior quality and control flexibility. Experimental results demonstrate the effectiveness of LLMs in augmenting DGMs via knowledge transfer and in facilitating multimodal data synthesis through collaborative interactions with diverse DGMs, which is further improved by the DM-based refinement process.