A CEP based ETL method of active distribution network operation monitoring and controlling signal data

With the development of Active Distribution Network, the scale of power system becomes larger and larger, and the number of electrical equipment in distribution network increases sharply and becomes further precise, electrical equipment's operation monitoring and controlling signal data has the characters of massive, diversity and complication, shows a trend of Big Data. Massive and random operation monitoring and controlling signal data causes various applications in active distribution network unable to extract useful information quickly and efficiently so as difficult to form decision support. Complex event processing (CEP) is an intelligent data processing technology rise in the era of Big Data, which can implement rapid analysis and processing to continuous data based on rule engine. The article uses CEP engine as the operation monitoring and controlling signal processing core, and uses ETL (Extract-Transform-Load) framework to integrate, clean and load the distributed, disordered and standard not unified signal data in active distribution network into the data warehouse. The problem of data format not unified and independent storage during data extraction can be solved by using the adapter mode and daemon process way. Based on CEP engine, it determines the core processing architecture of operation monitoring and controlling signal big data. In the architecture, signal cleaning rule library and algorithm library use the pluggable mode which makes them easy to maintain and expand. Rules library can be determined by using nested query, combined operation and pattern matching, and algorithm library can be packaged of memory partitioning and multithread processing, word-frequency statistics, keyword recognition and elimination, and other algorithms. It uses buffer queue to cache processing result and format the output as needed. The CEP engine based Big Data ETL solution implements the fast, accurate and effective standardization processing of operation monitoring and controlling signal and provides accurate data preparation for fast simulation, fault analysis, state estimation and other important application in active distribution network.