Kali: Scalable encryption fingerprinting in dynamic malware traces

Binary analysis of malware to determine uses of encryption is an important primitive with many critical applications, such as reverse-engineering of malware network communications and decryption of files encrypted by ransomware. The state of the art for encryption fingerprinting in dynamic execution traces, the ALIGOT algorithm — while effective in identifying a range of known ciphers — suffers from significant scalability limitations: in certain cases, even analyzing traces of a few thousands of machine instructions may require prohibitive time/space. In this work, we propose KALI, an enhanced algorithm based on ALIGOT which significantly reduces time/space complexity and increases scalability. Moreover, we propose a technique to focalize the analysis on encryption used for specific purposes, further improving efficiency. Results show that KALI achieves orders of magnitude reduction in execution time and memory utilization compared to ALIGOT, and processes real-world program traces in minutes to hours.