RANDART: A Hybrid Approach Leveraging File Traps and Registry Monitoring to Thwart Crypto Ransomware on Windows Endpoints

The rise of crypto-ransomware, especially targeting Windows systems leveraging multi-threading and swift encryption rates, challenges conventional detection methods, often resulting in delayed detection and significant file losses. Although trap file monitoring provides real-time detection, it may lead to false positives due to concurrent user activities on endpoints. Furthermore, there is a research gap in identifying ransomware processes, as most of the research focuses on real-time ransomware detection rather than identifying processes responsible for ransomware activity. To address the above limitations, we propose RANDART, a tool designed to detect and terminate ransomware activity in real-time while minimizing false positives and accurately identifying ransomware processes. RANDART achieves this through two distinct functional blocks: one monitors file modifications, while the other tracks modifications in the Windows Registry. The first block monitors file modifications, including delete, write, or rename operations on trap files. The trap files are selected using a non-parametric clustering method called Affinity Propagation paired with file access patterns of modern ransomware variants. The second functional block monitors critical registry hives in the Windows Registry for key additions, value additions, and value updates to confirm ransomware activity, thereby reducing false positives. Upon ransomware detection, RANDART employs a lightweight classifier, utilizing process attributes like I/O counters, page faults, handles, and threads as features to selectively identify and terminate ransomware processes among the running processes. In the evaluation, RANDART demonstrated an average latency of 1.7 seconds and a file loss rate of 0.35 % across 20 ransomware families, with no instances of false positives.