Agent multiplication: An economical large-scale testing environment for system management solutions

System management solutions are designed to scale to thousands or more machines and networked devices. However, it is challenging to test and verify the proper operation and scalability of management software given the limited resources of a testing lab. We have developed a method called agent multiplication, in which one physical testing machine is used to represent hundreds of client machines. This provides the necessary client load to test the performance and scalability of the management software and server within limited resources. In addition, our approach guarantees that the test environment remains consistent between test runs, ensuring that test results can be meaningfully compared. We used agent multiplication to test and verify the operation of a server managing 4,000 systems. We exercised the server functions with only 8 test machines. Applying this test environment to an early version of a real enterprise system management solution we were able to uncover critical bugs, resolve race conditions, and examine and adjust thread prioritization levels for improved performance.