POSITION PAPER: An Evaluation Framework for Scientific Programming Productivity

Substantial time is spent on building, optimizing and maintaining large-scale software that is run on supercomputers. However, little has been done to utilize overall resources efficiently when it comes to including expensive human resources. The community is beginning to acknowledge that optimizing the hardware performance such as speed and memory bottlenecks contributes less to the overall productivity than the development lifecycle of high-performance scientific applications. We propose an evaluation approach that compares recorded novice programming workflows to an expert workflow to identify productivity bottlenecks and suboptimal paths. We use these results to suggest improvements to the programming environment or tools. We give preliminary results from applying this approach to two case studies involving the use of numerical libraries.