Machine Learning-based Estimation of Story Points in Agile Development: Industrial Experience and Lessons Learned

Estimating story points is an important activity in agile software engineering. Story-point estimation enables software development teams to, among other things, better scope products, prioritize requirements, allocate resources and measure progress. Several machine learning techniques have been proposed for automated story-point estimation. However, most of these techniques use open-source projects for evaluation. There are important differences between open-source and commercial projects with respect to story authoring. The goal of this paper is to evaluate a state-of-the-art machine learning technique, known as Deep-SE [3], for estimating story points in a commercial project. Our dataset is comprised of 4,727 stories for a data anonymization product developed by a 27-member agile team at a healthcare data science company, IQVIA. Over this dataset, Deep-SE achieved a mean absolute error of 1.46, significantly better than three different baselines. Model performance nonetheless varied across stories, with the estimation error being larger for stories that had higher points. Our results further indicate that model performance is correlated with certain story characteristics such as the level of detail and the frequency of vague terms in the stories. An important take-away from our study is that, before organizations attempt to introduce machine learning-based estimation into agile development, they need to better embrace agile best practices, particularly in relation to story authoring and expert-based estimation.