A Multi-Modal Short-Term Photovoltaic Power Forecasting Framework

In the context of widespread solar energy deployment, accurate prediction of photovoltaic power generation is crucial to ensure safe and stable operation of power systems. Extracting spatiotemporal features from ground-based sky images can significantly improve the accuracy of ultra-short-term photovoltaic power forecasting. Although previous studies have relied heavily on multimodal data for power forecasting, multimodal datasets are usually small and may not be sufficient to fully train complex models, resulting in poor model performance. Directly processing multimodal data requires learning the relationship between multiple modalities simultaneously, which increases training complexity. In this paper, an innovative learning strategy is proposed to construct a more generalized deep learning model by pre-training the sky image processing module and the time series data processing module. Specifically, this paper first uses a single-modal dataset to train the image processing module and the time series reasoning module separately, and then fuses them into a multimodal framework to achieve prediction through interactive attention mechanism and adaptive cross-modal fusion. The results show that the components learned through single-modal training can effectively collaborate, allowing the multimodal framework to achieve competitive results after fine-tuning.