Index-based join operations in Hive

Indexing techniques are crucial for efficiency and scalability of processing queries over big data. Hive is a batch-oriented big data management engine that is well suited for data OLAP and data analysis applications. For very “selective” queries whose output sizes are a small fraction of the contributing data, the brute-force approach suffers from poor performance due to redundant disk I/O's or initiations of extra map operations. We make a first attempt and propose an index-based join technique to speed up the process and integrate it in Hive by mapping our design to the conceptual optimization flow. To evaluate the performance, we create and evaluate test queries on datasets generated using TPC-H benchmark. Our results indicate significant performance gain over relatively large data and/or highly selective queries having a two-way join and a single join condition.