Specification-based intrusion detection for H.323-based voice over IP

Voice over IP (VoIP) has been in a state of rapid development due to its economical advantage over traditional telephone services. Denial of service (DoS) attack has been a major security threat for many computer systems. This work introduces a specification-based intrusion detection system to protect H.323 gatekeepers from both external and internal DoS attacks. Based on the protocol for RAS (Registration, Admission Status) messages, a finite-state machine specification for correct behaviors between a gatekeeper and endpoints is produced. Security requirements against these DoS attacks are established, resulting in a formal protocol specification for secured gatekeepers. Developing the proposal into a practical solution, an intrusion detection module is built and incorporated into the open source software GNU Gatekeeper (also named OpenH323GK). A simple, proof-of-concept prototype has been built; the secured H. 323 gateway is able to fend off DoS attacks launched from GNU OpenPhone clients.