A process state-transition analysis and its application to intrusion detection

This paper describes a new technique for detecting security breaches in a computer system. For each Unix process, the user credentials, which are user identifiers, determine the process privilege, including whether a process has gained a high privilege, such as that of the superuser. The state transition technique is applied to a suitably defined process state, identified by certain classes of user credential values. A transition takes place when these values change from one class to another. These states are clearly defined, and prohibited state transitions as well as some supporting rules are identified. When many break-ins succeed, either the rules are violated or these prohibited transitions occur, and this implies a violation of system security policy. A specially modified system call, ktrace0, is used by the superuser to monitor the process-state and state transition analysis is applied to the traced information, by the Intrusion Detection System. Tests show that most known security violations belonging to the targeted classes (such as buffer overflow exploits) can be detected (and possibly pre-empted) while the constituent activities are still being processed in the kernel.