Efficiently Managing Large-Scale Keys in HDFS

To transparently encrypt/decrypt data stored in Hadoop Distributed File System (HDFS), the Key Management Server (KMS) has been designed to automatically assign encryption zone keys (EZK) to encryption zones. However, the current key-management scheme suffers from performance bottleneck. In this paper, dividing key files for storing key data into metadata file and material file, we propose a memory-disk coordinated key management architecture for Hadoop EZK to manage massive keys, where a tree structure is designed to index key metadata stored in memory, and the inverted index structure is used to record materials stored on hard disk. Further balance-tree-based management algorithms without Java serialization is designed to efficiently query keys (including metadata and materials). A series of theoretical analysis are conducted to show the upper and lower boundary of metadata query. Finally, we implement the above scheme in the Hadoop KMS without modifying the original key-management interfaces. A series of experiments are carried out on the simulated data. Experimental results show that, our scheme can efficiently manage ten billion keys and perform much better than the existing KMS in Hadoop.