Incorporating Deep Learning Model Development With an End-to-End Data Pipeline

The rising popularity of artificial intelligence has led to an increasing amount of research on deep learning models. Many current studies have focused on topics such as model structures, model optimization techniques, fine-tuning, and transfer learning, aiming to create novel models that have greater predictability in one or more fields of interest. However, while model development is important, it should not be limited to the topics mentioned above. Instead, the scope of research can be broadened to encompass the holistic design of an end-to-end pipeline for deep learning model development, which includes data storage, extract, transform, and load (ETL), business intelligence, model training and testing, and incremental learning. This paper therefore aims to underscore the importance of this data pipeline and provide a paradigm that delineates each aspect of this pipeline in detail through a practical case study centered on the end-to-end development of recommender system models. Compared to the conventional model development process, the novel data pipeline provides a more organized and efficient data storage and data preparation, an easier and more manageable visualization solutions, and a more comprehensive way for model evaluation and model selection through the usage of databases, business intelligence tools, and incremental learning.