ProtoMon: embedded monitors for cryptographic protocol intrusion detection and prevention

Intrusion detection systems (IDS) are responsible for monitoring and analyzing host or network activity to detect intrusions in order to protect information from unauthorized access or manipulation. There are two main approaches for intrusion detection: signature-based and anomaly-based. Signature-based detection employs pattern matching to match attack signatures with observed data making it ideal for detecting known attacks. However, it cannot detect unknown attacks for which there is no signature available. Anomaly-based detection uses machine-learning techniques to create a profile of normal system behavior and uses this profile to detect deviations from the normal behavior. Although this technique is effective in detecting unknown attacks, it has a drawback of a high false alarm rate. In this paper, we describe our anomaly-based IDS designed for detecting malicious use of cryptographic and application-level protocols. Our system has several unique characteristics and benefits, such as the ability to monitor cryptographic protocols and application-level protocols embedded in encrypted sessions, a very lightweight monitoring process, and the ability to react to protocol misuse by modifying protocol response directly.