Deep Learning Visualization for Underspecification Analysis in Product Design Matching Model Development

In machine learning model development, the best model for deployment is usually chosen among the models trained during hyperparameter tuning based on the performance on the test data. However, with the underspecification problem, this practice can be shortsighted as the solution may not perform well in the production environment. Hence, developers need a better tool to comprehend model results in both development and production environments to guide the next iteration of development. In this work, we present the novel visualization system for tracking model results with underspecification analysis in mind. In our evaluation, we show both qualitative and quantitative analysis of model development for product design matching problem and the results suggest that our method helps developers gain a better understanding of the models being developed, resulting in better performance in production. Finally, this visualization has been used at Wazzadu.com to help developers make decisions more effectively in model development.