Apache spark based urban load data analysis and forecasting technology research

Considering the complexity of data analysis caused by increasing urban load data, an Apache Spark based urban load analysis and forecasting technology is proposed in this paper. Firstly, the large-scale data processing platform is designed. Secondly, statistics of urban load data are processed by transformation and action in the form of resilient distributed datasets (RDDs). Thirdly, load forecasting model based on Dynamic Bayesian Network (DBN) is built for short-term urban load forecasting. GraphX is used for graph-parallel computation. Finally, the result of load analysis is presented and forecasting results of DBN are compared with other models. The comparison outcome shows that the DBN model implemented in Spark has better accuracy in load forecasting.