Detecting disguised processes using application-behavior profiling

In order to avoid detection, malware can disguise itself as a legitimate program or hijack system processes to reach its goals. Commonly used signature-based Intrusion Detection Systems (IDS) struggle to distinguish between these processes and are thus only of limited use to detect such attacks. They also have the shortcoming that they need to be updated frequently to possess the latest malware definitions. This makes them inherently prone to missing novel attack techniques. Misuse detection IDSs however overcome this problem by maintaining a ground truth of normal application behavior while reporting deviations as anomalies. In our approach, we try to accomplish this by observing a process' memory consumption. This is for two reasons: We expect the readings to be less volatile in comparison to for instance network operations. Second, by breaking the problem down, we are able to investigate thoroughly while still laying the foundations for future expansion. We use the observations from a given host to train a machine learning algorithm. After an initial learning phase, we evaluate the model with readings from the application it has been trained on and other applications in order to assess its quality. Our results indicate that the efficacy of this method is highly dependent on parametrizing the machine learning algorithm appropriately. A large variance in accuracy with only slightly altered inputs confirms this suggestion. We finish with a discussion on deploying such an IDS at scale in a realistic scenario.