ModelKB: Towards Automated Management of the Modeling Lifecycle in Deep Learning

Deep Learning has improved the state-of-the-art results in an ever-growing number of domains. This success heavily relies on the development and training of deep learning models, also known as deep neural networks (DNN). Often, developing a DNN is an ad-hoc, iterative process that results in producing tens to hundreds of models before arriving at a satisfactory result. While there has been a surge in the number of tools and frameworks that aim at facilitating deep learning, the issues of model management have been largely ignored. In particular, deep learning practitioners have to manually track their experiments using text files, spreadsheets or folder hierarchies, which is expensive, time-consuming, and error-prone. In this paper, we present our ongoing work and vision towards automating end-to-end model management in deep learning. Specifically, we introduce a tool prototype, named ModelKB, that can automatically (1) extract and store the model's metadata-including its architecture, weights, and configuration; (2) visualize, query, and compare experiments; and (3) reproduce experiments. Our overarching goal is to automate the model management process with minimal user intervention using the user's favorite framework. We report the current status of ModelKB, a pilot user study, and the challenges of automating model management in deep learning.