Improved kernel security through memory layout randomization

The vast majority of hosts on the Internet, including mobile clients, are running on one of three major operating system families. Malicious operating system kernel software, such as the code introduced by a kernel rootkit, is strongly dependent on the organization of the victim operating system. Due to the lack of diversity of operating systems, attackers can craft a single kernel exploit that has the potential to infect millions of hosts. If the underlying structure of vulnerable operating system components has been changed, in an unpredictable manner, then attackers must create many unique variations of their exploit to attack vulnerable systems en masse. If enough variants of the vulnerable software exist, then mass exploitation is much more difficult to achieve. Many forms of automatic software diversification have been explored and found to be useful for preventing malware infection. Forrest et. al. make a strong case for software diversity and describe a few possible techniques including: adding or removing nonfunctional code, reordering code, and reordering memory layouts. Our techniques build on the latter. We describe two different ways to mutate an operating system kernel using memory layout randomization to resist kernel-based attacks. We introduce a new method for randomizing the stack layout of function arguments. Additionally, we refine a previous technique for record layout randomization by introducing a static analysis technique for determining the randomizability of a record. We developed prototypes of our techniques using the plugin architecture offered by GCC. To test the security benefits our techniques, we randomized multiple Linux kernels using our compiler plugins. We attacked the randomized kernels using multiple kernel rootkits. We show that by strategically selecting just a few components for randomization, our techniques prevent kernel rootkit infection.