Tengu: A Distributed Audit Log Storage System

Nowadays enterprises and government organizations are facing a great threat from advanced persistent threats (APT), which has become one of the most dangerous challenges in recent years. Intrusion detection methods based on provenance graph have been considered as the key to APT defense in the future. As system logs capture complex causal dependency relationships between system entities, they have become the main data source for constructing provenance graphs. However, as modern computer systems become more and more complex, system logs may be accumulated in large quantities. Existing attack detection methods mostly adopt a centralized detection architecture, which sends all system audit logs to the server for processing, incurring prohibitive costs in data transfer, data storage and computation. To address the above fundamental challenges, we propose Tengu, a distributed audit log storage system that can reduce the space occupied during log storage process by a distributed approach. Our system is evaluated ona large public dataset, and experimental results show that our system improves compression performance by 66% compared with existing log compression methods.