System abnormality detection in stock market complex trading systems using machine learning techniques

Stock market trading systems are real time systems that process thousands of data per minute and are considered to be critical as well as complex. It incorporates the features of complex business processing and sophisticated in-memory processing techniques for speed and throughput. These systems are distributed in nature, and they use a large number of processing nodes incorporating fault tolerance mechanisms. Complex systems also have a large effective number of strongly interdependent variables. Hence detecting faults and failures in stock market systems is a complex and cumbersome task. The study explores machine learning techniques to detect anomalous behavior to provide warnings before a system results in a fault or failure state. The study extensively utilizes a supervised learning approach with machine learning algorithms such as C4.5, Naïve Bayes, and ensemble techniques; bagging and Random Forest. The system statistics captured from log files are preprocessed and transformed to eliminate system environment dependencies. For each of the three components the initial feature selection is carried out manually using domain knowledge and expertise. Initial feature selection based on domain expertise was required as the number of features per component is large and does not closely relate to the system state. Feature selection methods (Info Gain algorithm with Ranker search) have been successfully employed to filter out unrelated attributes and to reduce computational complexity. A comparative evaluation is performed under each component status prediction. This study also utilizes oversampling techniques to overcome limitations caused by the class imbalance phenomena. A range of evaluators are used to analyze the results and effectiveness of the models. The highest accuracy and Receiver Operating Characteristic (ROC) values are achieved when C4.5 decision tree is applied to the oversampled feature set and when the Random Forest algorithm is applied to the oversampled feature set. However, precision, recall and F-measure values vary. Root cause detection for anomalies and numeric values for system health predictions are future work in the research.