Harmonizing Recurrent and Attention Mechanisms: The Fused Gate Model

Modern neural architectures have strongly favored global self-attention mechanisms for sequence modeling, discarding the enduring effectiveness of Long Short-Term Memory networks—particularly Bidirectional variants (BiLSTMs)—in capturing critical local dependencies. While self-attention excels at modeling global interactions, this shift overlooks BiLSTMs’ capacity to resolve local nuances like syntactic parsing or negation handling, which remain pivotal for tasks such as sentiment analysis and named entity recognition. In this work, we introduce DGRAN (Dynamic Gated Recurrent-Attention Network), a novel hybrid architecture that harmonizes the synergy between BiLSTMs and Sparse Attention mechanisms. DGRAN dynamically fuses local feature extraction (via BiLSTMs) and global pattern modeling (via window-restricted sparse attention) through two innovations: a context-aware fusion gate that adaptively balances these representations, and inter-layer feedback for iterative refinement. The design ensures linear complexity, enabling scalability to long documents. Evaluated on textual modeling tasks, DGRAN achieves 83.64% validation accuracy on arXiv classification, surpassing BiLSTM-only (62.88%) and sparse attention-only (81.52%) baselines, with smoother convergence. This work opens the door for harmonized architectures in NLP, grounding the way for applications demanding rich contextual understanding with fine grained reasoning.