Query Optimization for Air Quality Big Data based on Hive-ORC

To improve the efficiency of analyzing the massive amount of monitoring data collected from an air quality monitoring system, a method based on Hive data warehouse to store data as ORC file format then create Row Group Index and Bloom Filter Index is proposed to optimize the query of air quality big data. The air quality monitoring data from the environmental monitoring center of Hubei province was taken as a research object. After the data was transferred to Hive, Spark was used to query and analyze data. Five queries were carried out on three datasets to conduct the comparison experiment between the method proposed in this paper and the method of storing data as TextFile by default on Hive. The results show that the optimization method based on Hive-ORC and its indexes reduces the storage space of big data by 90% and reduces its query time by 86%. What's more, better optimization effect can be achieved with the increase of data volume.