Recent Progress in Understanding Transformer and Developing Its Surpasser: A Survey

Transformers have significantly impacted machine learning, particularly in natural language processing and computer vision, due to their robust attention mechanisms and scalability. Although it is very successful and has the attention mechanism to explain it, it is still a black box in general. This lack of transparency is not conducive to applications, nor is it conducive to the development of more advanced models. We systematically analyze current methodologies for interpreting the attention patterns, hidden representations, and decision-making processes within Transformers. Additionally, we investigate how these insights aid in refining Transformer architectures and inspire the creation of innovative models that extend beyond the conventional Transformer frameworks. Due to its expertise in long-term dependencies and data scalability, Transformer undoubtedly has great advantages in foundation language models, achieving multimodality, and processing discrete data, but it is inferior to Mamba in terms of efficiency, especially when processing continuous data. It is inferior to Diffusion Model when processing image data because it is not good at grasping the distribution of high-dimensional data and is relatively less suitable for image generation model. By integrating achievements in both explaining and advancing Transformer-like architectures, this paper serves as a valuable resource for researchers aiming to enhance the performance, transparency, and efficiency of Transformers in various applications and to develop models that surpass current Transformer paradigms.