Asynchronous Index Strategy for high performance real-time big data stream storage

Big data insert-intensive applications challenge traditional RDBMS. Key-Value databases achieve the same throughput with much more price/performance ratio, which makes them popular recent years. However, Key-Value databases are not suitable for high performance real-time applications. In this paper we introduce Asynchronous Index Strategy as a high performance solution for insert-intensive time series big data storage. It takes advantage of partial replication and asynchronous indexes, which results in zero overhead for index updates. Furthermore, a general middle-ware for clustering databases based on Asynchronous Index Strategy is implemented. Finally, indexing and inserting performance experiments highlight the efficiency of Asynchronous Index Strategy. As for AIS based on MongoDB, it achieves a throughput that is 17 times of MongoDB sharding cluster.