Efficient Classification and Rapid Processing of Big Data in Power Distribution Networks

Grid data management technology has taken as a pivotal role in power system operation. However, with escalating complexity and scale, the effective management of grid data has emerged as a formidable challenge. In this study, a method for the rapid processing and classification of massive grid data is proposed, comprising two primary components: data pattern definition and data processing and querying. Data pattern definition encompasses identifying repetitive, sparse, and increasing/decreasing patterns. We use the fast Fourier transform for autocorrelation coefficients calculation to define the repetitive score, and the sparse score and increasing/decreasing score are also defined for evaluating three patterns. Three different dimension reduction methods, viz. PCA, t-SNE and UMAP and data without dimension reduction are combined with K-means clustering for exploring the potential distribution of samples. Subsequently, variational autoencoder (VAE) is used to extract the hidden features of the data and the dimension reduction methods are applied to the hidden structure of data obtained from VAE. After feature extraction via VAE and dimensionality reduction using UMAP, combined with K-means clustering, five cluster centers were obtained with a silhouette coefficient of 0.85. Further analysis incorporating repetitive score, sparse score, and increasing/decreasing score reveals that the five clusters represent repetitive patterns, sparse patterns, increasing patterns, decreasing patterns, and normal patterns within the data.