PoolRunner: An Extensible Performance Testing Simulation Tool for Thread-Pool Middleware

Thread Pool System (TPS) is the most dominant concurrency control middleware service that boosts the scalability and performance of multithreaded servers. However, the performance evaluation of TPS is problematic. The performance of a TPS is tested at the end of the project by the load testing of the multithreaded server (in which the TPS is embedded) in a testbed environment. The TPS cannot be assessed alone early in the SDLC at the design time of the multithreaded server. This lacks the notion of early performance testing. Furthermore, the testbed construction and configurations are complex, difficult, time-consuming, and expensive, especially for researchers. Setting up the lab, learning testbed configurations, embedding TPS in an open-source multithreaded application (e.g. a web server or an application server), selection of load testing tool, and getting expertise to use the tool; all these are added costs for a researcher. Moreover, the load testing tools in the testbed environment cannot assess the internal performance metrics of TPS because the tool communicates only with the multithreaded server at the application layer. Thus, in this paper, we present the PoolRunner, a client-server-based extensible performance testing simulation tool for TPS, developed in JAVA. The PoolRunner is required not only by the practitioners in the industry in early performance testing when architectural choices are made for multithreaded servers, but also by researchers in the academia to assess and compare the performance of their new TPS with existing state-of-the-art TPSs. To the best of our knowledge, there is currently no publicly available simulation tool specifically designed for the early-stage performance evaluation of TPS. Our work contributes by developing a TPS-based performance testing simulation tool and by providing an extendible framework within the tool that enables researchers to embed and compare their custom TPS implementations with existing ones. As an example, we have embedded in the PoolRunner a state-of-the-art TPS of Oracle corporation as a baseline, as well as a static TPS for learning and validity. A detailed experimental evaluation of PoolRunner validates the correctness, robustness and high performance of PoolRunner.