Statistical Profiling of n-grams for Payload Based Anomaly Detection for HTTP Web Traffic

Anomalous HTTP traffic can be identified by analysing the content of HTTP packet as payload. n-gram analysis is a prominent technique for payload analysis. In this paper, a novel n-gram based anomaly detection method has been proposed for HTTP traffic. During the training phase, statistical profiling (the maximum, the minimum, the median and the average of number of occurrences in a packet) of n-grams for a data set of normal (not malicious) HTTP packets provides the basis for this work. In a test packet, the number of occurrences of an n-gram decides whether the n-gram is anomalous or not. Moreover, the deviation of number of occurrences of such an anomalous n-gram from the median (or the average) of number of occurrences of the n-gram in training packets is considered for estimating an anomaly score of the test packet. Consideration of this magnitude of the deviation from the statistical profile (median or average) of n-gram occurrences for a normal HTTP traffic is the highlight of the proposed method. Finally, an anomaly-to-normal ratio for the test packet determines whether it is malicious or normal. This technique yields better performance as compared to an existing n-gram based method of anomalous HTTP traffic detection.