LLM-Augmented Multi-Fidelity Bayesian Optimization for Parameter Optimization in Human-Robot Collaborative Assembly

Human-robot collaborative assembly (HRCA) is critical for manufacturing tasks like aircraft cabin drilling, where human contextual adaptability complements robotic repeatability. Parameter optimization—a critical component of HRCA systems—faces significant challenges due to limited high-fidelity experimental data, primarily resulting from costly physical trials and dynamic human-robot interaction uncertainties. While large language models (LLMs) show promise as low-cost estimators for parameter-quality relationships, their unreliable physical reasoning and hallucination-prone outputs limit direct application in safety-critical scenarios. To address these challenges, this paper proposes an LLM-augmented multi-fidelity Bayesian optimization framework for parameter optimization. This approach leverages LLMs’ strengths while mitigating the scarcity of real samples and minimizing potential hallucination issues. First, a GPT-based LLM serves as the low-fidelity model, generating initial parameter-quality predictions using tailored prompts that encode domain-specific physics. Then, a latent variable Gaussian process (LVGP) hierarchically links the LLM’s predictions with high-fidelity simulations through a shared covariance structure, enabling residual-based uncertainty propagation. Following this framework, an adaptive acquisition function prioritizes parameter combinations that maximize cross-fidelity consensus. Simultaneously, deviations between LLM and high-fidelity outputs trigger residual-guided prompt engineering to refine the LLM’s reasoning. Failed optimization trials are systematically accumulated to iteratively retrain the LVGP kernel and adjust the LLM’s prompting strategy. A case study of parameter optimization for robotic arm drilling in aircraft cabin assembly will demonstrate the advantages of the proposed method.