How do we know that management is working?

Over the last years, the Management community has witnessed a shift away from information models and protocols towards value-added management services that improve the configuration and fault management of a distributed system, or optimize its performance. However, whenever we try to articulate the value of Management, it turns out that we neither have the methodologies, nor the tools to help us assess where we as a discipline are on the maturity curve and how 'self-managing' the systems we build actually are. Metrics such as 'total cost of ownership' or 'number of servers per administrator' are often overly simplistic and essentially focus just on symptoms, not on the true factors that impact the value of management. It is therefore hard, if not impossible, to quantify the value that the investment in management technology actually yields actually yields, and there are no hard metrics available that facilitate the comparison between management systems from different vendors. The panel will address the following issues: 1. Can we measure automation and are we able to assess its value? 2. Is there a way to develop a 'Capability Maturity Model' for Management? 3. What would such a model look like? 4. What are the key performance indicators of Management? 5. What lessons can we learn from system benchmarks that have been developed over the last 15 years? 6. Will we ever see TPC-Management or SPECManagement benchmark suites?