Application of Signal Detection and Estimation Theory to Network Security

The need to use quantitative methods to detect intrusion is increasing due to the high false positive and false negative rates of existing Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS). Most network security techniques employed by the IDS and IPS depend mainly on packet behavior for detection. This work applies a quantitative approach based on Maximum A Posteriori (MAP) detection rules with the hope of reducing the high false positive and false negative rates. The entire system has been represented by a mathematical model of a discrete binary communication channel having two possible input messages and two possible output symbols. The network under study is assumed to have only one entry point (sender) for now, with a number of nodes (receivers). Also, all normal operational packets are referred to as normal packets and any other packets are referred to as abnormal packets. The analysis strategy used here is anomaly detection. The developed algorithm initially calculates the a priori probabilities for the normal and abnormal packets both at the sender and entry ends. These values are further used in finding the threshold probabilities to be compared to the corresponding probabilities of future incoming packets. MATLAB was used in coding the developed algorithm. This work will be expanded by modeling the entire system as a continuous binary communication channel and also by considering multiple entry points as future works, with the intension of improving the results obtained so far.