Machine learning product key performance indicators and alignment to model evaluation

Machine Learning has seen amazing progress the past years with increasing commercial use from industries across the business spectrum. Businesses strive for alignment of vision and mission statement to the actual products they sell. For that reason tools like the Key Performance Indicators exist in order to monitor such progress. Nevertheless, products that embed a machine learning component are being optimized with other objective functions and are being evaluated in a vacuum with specific performance evaluation metrics that often have nothing to do with the business vision. In this position paper, we highlight this gap in different instances of the machine learning life cycle, explore and critically evaluate the current available solutions in the literature and introduce Key Performance Indicators in the machine learning development process. The paper also discusses representative machine learning KPIs in the development and deployment process.