Measuring Fidelity and Utility of Time Series Generative Adversarial Networks

Generative Adversarial Networks (GANs) have emerged as tools for creating synthetic data that mimics real datasets. Time series GANs extend GANs concept by attempting to replicate the temporal dependencies and patterns of time series, while preserving the privacy of the real data. However, measuring the fidelity and utility of a synthetic time series remains a challenge. Thus, this paper discusses the pros and cons of quantitative metrics to assess synthetic time series generated by GANs based on sensitive data of two application domains. We first review how metrics based on probability distributions, such as Kullback-Leibler Divergence, Jensen-Shannon Distance, Wasserstein Distance, and Maximum Mean Discrepancy, have been used to evaluate synthetic data. Meanwhile, to assess the utility of synthetic data the Testing on Synthetic, Training on Real (TSTR) score is used. To assess these metrics, we compare three time series GANs (RGAN, TimeGAN, Doppelganger) to generate synthetic datasets of two network domain applications: i) user content requests for a Brazilian streaming provider; and ii) devices connected to mobile base stations in a large Brazilian city. We compare the fidelity of synthetic datasets and assess their utility in predicting content requests and the number of users connected to a given mobile base station.