Statistical Profiling of n-grams for Payload Based Anomaly Detection for HTTP Web Traffic

Anomalous HTTP traffic can be identified by analysing the content of HTTP packet as payload. n-gram analysis is a prominent technique for payload analysis. In this paper, a novel n-gram based anomaly detection method has been proposed for HTTP traffic. During the training phase, statistical profiling (the maximum, the minimum, the median and the average of number of occurrences in a packet) of n-grams for a data set of normal (not malicious) HTTP packets provides the basis for this work. In a test packet, the number of occurrences of an n-gram decides whether the n-gram is anomalous or not. Moreover, the deviation of number of occurrences of such an anomalous n-gram from the median (or the average) of number of occurrences of the n-gram in training packets is considered for estimating an anomaly score of the test packet. Consideration of this magnitude of the deviation from the statistical profile (median or average) of n-gram occurrences for a normal HTTP traffic is the highlight of the proposed method. Finally, an anomaly-to-normal ratio for the test packet determines whether it is malicious or normal. This technique yields better performance as compared to an existing n-gram based method of anomalous HTTP traffic detection.

I. Introduction

Intrusion detection system (IDS) is an indispensable tool in an organization’s armour to protect it from cyber attacks. An IDS can be classified as signature based or anomaly based. In signature based systems, already known signatures of attacks are stored in a database. If any of these attack signatures is found in an incoming packet, it is treated as malicious. The advantage of this type of IDS is that a malicious packet can be accurately detected if its signature is already known. The disadvantage of this type of IDS is its inability to detect zero day attacks. Examples of popular signature based IDS are BRO [1] and SNORT [2] . On the contrary, the anomaly detection method models the normal network behavior, and identifies anomalies as deviations from normal network behavior. They are appealing because of their ability to detect previously unseen attacks. A good survey of intrusion detection systems can be found in [3] .