Hunting for Malware Code in Massive Collections

A binary program can be abstracted as a set of hashes computed on pieces of disassembled code. Associating these pieces of code with other programs can provide extra information for malware researcher, useful for detection, attribution and for determining the Indicators of Compromise. Current methods for searching files that contain a given piece of code provide poor performance. The paper presents an original solution which is efficient, providing a scalable approach at the same time. The code hashes are indexed using the inverted index, which is designed and implemented as an in-memory lookup table, providing constant time for getting the files that contain a code hash and is optimized for storage efficiency. The lookup table associates a code hash to the files that contain it. Furthermore, our solution offers support for computing code similarity, which is achieved by using, along with our indexing mechanism, a traditional NoSQL database that stores the forward index (association between a file and its code hashes) and the file metadata. Our solution provides fast update capability, due to the batch operations proposed when adding new files. When it comes to building the inverted index, our solution is 50 times faster than MongoDB given a collection of one million samples.