Dimensionality Reduction for Smart Meter Data Using Variational Autoencoders

The growth of smart grid technology has led to the generation of exceptionally high-dimensional energy datasets, owing to the increased frequency of measurements. Moreover, utilizing a year-long dataset is imperative, as relying solely on monthly or quarterly data may lead into wrong conclusions due to variations in seasonal dynamics and customer behavior. In order for this vast amount of information to be usable, it must be summarized into a low-dimensional representation. In this paper, we propose a Long Short Term Memory-based Variational Autoencoder (LSTM-VAE) to generate an encoded representation of the original complex multivariate data. By utilizing this method, a 8760-dimensional dataset can be condensed into a 10-dimensional dataset, which can be subsequently further reduced by Principal Component Analysis (PCA) to a 2D representation for visualization purposes. In addition to visually inspecting the 2D representation of the data, the effectiveness of this method is assessed by a subsequent clustering algorithm using the Adjusted Rand Index (ARI). Both visual inspection and the ARI index demonstrate that the proposed method outperforms other dimensionality reduction techniques by a substantial margin.