Improving long text classification based on Selective State Space model (Mamba)

Recently, with the use of transformer-based models, a big progress has been achieved across many areas of Natural Language Processing (NLP) including text classification. However, such progress is often limited to the short sequences as the standard approach with self-attention becomes slow and inefficient for long sequences due to its quadratic complexity. As a new architecture, Mamba addresses this challenge using Selective State Space Models (SSMs). Unlike previous SSMs, Mamba empowers its parameters to be input-dependent, enabling content-aware information propagation along the sequence. Also, Mamba integrates these selective SSMs into a streamlined architecture, achieving significant speedup and linear scaling with sequence length. In this paper, by taking the advantage of the smallest pre-trained Mamba model, Mamba-130, our Mamba 130m classifier has achieved a balance between the efficiency and the performance making it ideal for initial investigation. Although Mamba's efficiency and effectiveness empower it to handle sequences up to 1 million elements long, eliminating the need for truncation or chunking within documents, but we are limited to only use 15 GB of VRAM on Google Colab Pro and employ a constrained sequence length to address memory limitations. However, the experimental results presented in this paper show that our Mamba 130m classifier still outperforms the state-of-the-art approaches on long text classification tasks across multiple benchmark datasets.