Long-context Transformers: A survey

Current Transformer architecture falls behind in terms of the computational complexity of its attention mechanism, namely being O(N²). That problem leads to this model being unable to efficiently work with long sequences. There were a lot of attempts which tries to deal with this problem. In this paper, we considered the most up-to-date methods and how these methods compare to each other in different tasks. We address their limitations and how well they perform, and which ones are most versatile and could be used as a base for future works.