Quantitative Analysis for Non-linear System Performance Data Using Case-Based Reasoning

Effective software architecture evaluation methods are essential in today's system development for mission critical systems. We have previously developed MEMS and a set of test statistics for evaluating middleware architectures, which proven an effective assessment of important quality attributes and their characterizations. We have observed it is common that many system performance response data are not of linear nature, where using linear modeling is not feasible in these scenarios for system performance predictions. To provide an alternative quantitative assessment on the system performance using actual runtime datasets, we developed a set of non-linear analysis procedure based on Case-based Reasoning (CBR), a machine learning method widely used in another disciplines of Software Engineering. Experiments were carried out based on actual runtime performance datasets. Results confirm that our non-linear analysis method CBR4MEMS produced accurate performance predictions and outperformed linear approaches. Our approach utilizing CBR to enable performance assessments on non-linear datasets, a major step forward to support software architecture evaluation.