TSB-Forecast: A Short-Term Load Forecasting Model in Smart Cities for Integrating Time Series Embeddings and Large Language Models

In smart cities, energy management systems are essential for efficient resource utilization, enhanced operational efficiency, and sustainability promotion. This work presents a novel load forecasting model, TSB-Forecast (Time Series BERT), a hybrid machine learning model aiming to improve short-term electrical demand forecasting by integrating structured and unstructured data. The model uses Sentence Bidirectional Encoder Representations from Transformers (SBERT) to extract semantic characteristics from textual news and Time to Vector (Time2Vec) to capture temporal patterns, acquiring cyclical behavior and context-sensitive impacts. TSB-Forecast shows significant improvements reducing MAE by 38.7%, RMSE by 18.2%, and SMAPE by 50.6% compared to several baselines, including the official ENTSO-E forecast, Extra Trees Regressor (ETR), a hybrid model with basic text processing (M6), and a deep learning model combining GRU and CNN (CNN-GRU). TSB-Forecast demonstrates a stronger ability to combine time-related and contextual information, which helps reduce errors. Our contributions in this paper are fourfold and can be summarized as follows: The model initially integrates multi-source inputs, encompassing load demand, BBC News, meteorological data, and holiday information. Secondly, it utilizes Time2Vec and SBERT for improved temporal and semantic feature encoding. Subsequently, Extreme Gradient Boosting (XGBoost) and Extra Trees Regressor (ETR) collectively enhance the robustness of a stacked ensemble. Finally, a thorough empirical analysis showing consistent outperformance of the proposed model over all the forecasting measures against baseline models.