Modeling program behaviors by hidden Markov models for intrusion detection

Intrusion detection is an important technique in the defense-in-depth network security framework and a hot topic in computer network security in recent years. In this paper, a new efficient intrusion detection method based on hidden Markov models (HMMs) is presented. HMMs are applied to model the normal program behaviors using traces of system calls issued by processes. The output probability of a sequence of system calls is calculated by the normal model built. If the probability of a sequence in a trace is below a certain threshold, the sequence is flagged as a mismatch. If the ratio between the mismatches and all the sequences in a trace exceeds another threshold, the trace is then considered as a possible intrusion. The method is implemented and tested on the sendmail system call data from the University of New Mexico. Experimental results show that the performance of the proposed method in intrusion detection is better than other methods.