Using piecewise polynomial activation functions and relevance attention for long-term time-series prediction

Because of the introduction of self-attention mechanisms, various variants of Transformers have demonstrated significant potential for applications in time-series forecasting in recent years. This mechanism enhances the ability of the model to effectively capture correlations at different positions in a sequence, thereby endowing it with powerful global perception capabilities. However, traditional attention computation methods face challenges in accurately calculating attention scores and lack the flexibility required to model complex relationships. To address these issues, we propose a relevance attention mechanism. This mechanism emphasizes the model’s perception of input variations, enabling it to focus flexibly on different features and effectively perform global feature extraction. Unlike convolutional neural network (CNN) structures, the self-attention mechanism cannot selectively model local features. Therefore, we introduce an adaptive multiscale feature fusion module based on CNNs specifically designed to model local features. This module combines local features with global correlations to capture the overall characteristics of the time series more effectively. Finally, considering the limitations of existing activation functions such as Rectified Linear Unit with a leak (LeakyReLU), which exhibit issues such as C⁰ continuity and neuron death, we adopt a piecewise polynomial function, called C² piecewise polynomial activation (PPA), to provide a more robust and effective activation mechanism for neural network training. Our proposed model, which uses piecewise polynomial activation functions and relevance attention for long-term time-series prediction, outperforms existing models on six benchmark datasets, providing robust support for the further advancement and practical applications in the field of long-term time-series forecasting.