The Implementation of a Data-Accessing Platform Built from Big Data Warehouse of Electric Loads

With the flourishing of Internet of Things (IoT) technology, ubiquitous power data can be linked to the Internet and be analyzed for real-time monitoring requirement. Numerous power data would be accumulated to even Terabyte level as the time goes. To approach a real-time power monitoring platform on them, an efficient and novel implementation techniques has been developed and formed to be the kernel material of this thesis. The generic power-data source is provided by the so-called smart meters equipped inside factories located in an enterprise practically. The data collection and storage are handled by the Hadoop subsystem and the data ingestion to Hive data warehouse is conducted by the Spark unit. On the aspect of system verification, under single record query, these software modules: Hive, Spark, and Impala had been tested in terms of query-response efficiency. And for the performance exploration on the statistical query function and data ETL processing. The kernel contributions of this research work can be highlighted by two parts: (1) Multi-layer software modules are adopted to design and implement the real-time power-monitoring platform embedded with some excellent characteristics of high efficiency, high feasibility and low cost. (2) The rudimental experiments are conducted to verify the query-response efficiency, and performance evaluations for the proposed real-time power monitoring platform, which reveals the high feasibility for the target research goals.