A cooperative botnet profiling and detection in virtualized environment

Cloud security becomes an important topic in recent years, as to overcome the botnet in a virtualized environment is a critical task for the cloud providers. Although numerous intrusion detection systems are available, yet it is not practical to install IDS in every virtual machine. In this paper, we argue that a virtual machine monitor (VMM) can support certain security functions that our proposed design can actively collect information directly from the VMM without installing an agent in the guest OS. In addition, bot could not aware of the existence of such detection agent in the VMM. The proposed detection mechanism takes both passive and active detection approaches that the passive detection agent lies in the VMM to examine the tainted data used by a bot to check against bot behavior profiles and the active detection agent that performs active bot fingerprinting can actively send specific stimulus to a guest and examine if there exists expected triggered behavior. In the real-world bot experiments, we show the passive detection agent can distinguish between bots and benign process with low false positive and false negative rates. Also, the result shows the active detection agent can detect a bot even when before it performs its malicious jobs. The proposed mechanism suites an enterprise having cloud environment well to defeat malware.