Designing an Interpretable and Efficient AutoML Pipeline for Enhanced Data Analytics

This work proposes an automated machine learning (AutoML) pipeline that streamlines critical processes, including data preprocessing, feature engineering, text analysis, and model interpretability. This approach leverages Deep Feature Synthesis (DFS) for automated feature generation. It utilizes Large Language Models (LLMs) to transform textual data into compact, predictive categorical representations, reducing dimensionality while preserving semantic relevance. Robust preprocessing techniques—such as data type adjustments, imputation, encoding, and normalization—enhance data quality, while Bayesian Optimization facilitates efficient hyperparameter tuning. In addition, integrating LIME provides transparent insights into model predictions and feature selection processes, thereby fostering trust in decision-making. Evaluations on diverse datasets from healthcare, education, telecommunications, and finance demonstrate significant performance improvements, with key metrics increasing by over 10%. This pipeline effectively addresses the challenges of integrating textual data into structured formats and optimizing model parameters, supporting more effective data-driven decision-making.