Anomaly Detection in Resource Constrained Environments With Streaming Data

Isolation Forest (or
iForest
) is a well-known technique for anomaly detection. It is, however, a bulky approach that assumes the luxury of large storage space and is also ineffective with dynamic streaming data so common nowadays in varied application domains. In this work, we present the Preprocessed Isolation Forest (
PiForest
) approach for anomaly detection that works well in resource constrained environments and is also effective on streaming data.
PiForest
is largely based on the
iForest
algorithm and to effectively handle the streaming data includes a pre-processing stage. In the pre-processing stage, Principal Component Analysis (PCA) is first harnessed to significantly reduce the dimension and bulk of the data. Subsequently, the streaming characteristic of the data is handled through a sliding window mechanism that creates sequential blocks of data for systematic processing.
PiForest
is able to identify anomalies as effectively as
iForest
and other state-of-the-art anomaly detection techniques but has substantially low storage and prediction complexity. We conduct empirical evaluation of the proposed approach with standard data sets and show that it performs comparably with standard techniques in terms of Area Under the Receiver Operating Characteristic Curve (AUC-ROC) and is able to work with high-dimensional, streaming data. Subsequently, we do a real-world hardware implementation of
PiForest
and demonstrate that the approach is realistic and practicable in resource-constrained environments.