How to measure useful, sustained performance

Sustained performance is the amount of useful work a system can produce in a given amount of time on a regular basis. How much useful work a system can achieve is difficult to assess in a simple, general manner because different communities have their own views of what useful work means and because a large number of system characteristics influence its usefulness. Yet, we, as a community, intuitively, and sometimes explicitly, know when a system is more useful than another. On the other hand, we also know when measures do not accurately portray a system's usefulness. This report will review the important concepts of measuring sustained performance, discuss different approaches for doing the measurements and point out some of the issues that prevent our community from developing common, effective measures.