Incremental Outlier Detection in Air Quality Data Using Statistical Methods

Air pollution is one of the biggest problems being faced by the world today and effective measures are being sought to counter this menace. In order to devise a proper strategy of countermeasures, it is imperative to ensure that the data on which the modeling and decision making is being based is clean. Outliers usually creep in the sensor generated data due to human and system errors and have a degrading effect on the decision making. A number of methods have been tested for detection of outliers but an incremental approach is highly sought after as the temporal air quality data is available as a data stream. We have proposed a framework to analyze the performance of statistical outlier detection methods as being the answer to the continuous analysis problem. We have presented a comparative analysis of five statistical methods viz. Z-score, InterQuartile Range, Grubb’s Test, Hampel’s test, Tietjen-Moore Testfor outlier detection when used over the complete dataset and when used in an incremental mode.