LLM Prompting Strategies in Automated Machine Learning

Large Language Models (LLMs) present a promising way to include expert knowledge when building machine pipelines. This paper presents a study on how to integrate LLMs with Automated Machine Learning (AutoML) systems and empirical metadata from earlier experiments. One contribution of our work was comparing two similarity metrics in datasets to leverage relevant prior metadata to improve LLM-guided pipeline generation. In addition, we evaluated four levels of challenge in the LLM through prompts to identify boundaries. Likewise, we explore the use of LLM as a further optimizer for Bayesian optimization. We identify and mitigate potential pitfalls, delineate the current potential and limitations of LLMs in this setting, and provide valuable insight into LLM-guided pipeline generation methods in AutoML and the industry.