Weird-Net: Weighted Relative Distance Attention for Efficient and Robust Sequence Processing

Sequence processing is a fundamental research area in artificial intelligence (AI) that encompasses various tasks and applications. Existing models—such as recurrent neural networks (RNNs) and transformers—have drawbacks such as slow computation, high complexity, and overfitting. In this article, we propose Weird-Net, a novel sequence processing model that leverages the weighted relative distance (Weird)-attention mechanism. Weird-Net can capture positional inductive relationships more robustly and efficiently than transformers. Moreover, it enables parallel computation and can handle extremely long sequences with near-linear complexity. We conduct extensive experiments on multiple datasets and tasks and evaluate Weird-Net using various metrics. The results demonstrate that Weird-Net achieves state-of-the-art (SOTA) performance on several language modeling benchmarks and surpasses other models in terms of accuracy, speed, and memory usage.