The Advance of Deep Learning and Attention Mechanism

Due to the significant increment of computing power in recent years, research related to machine learning developed and changed dramatically. A new learning model named Transformer, based on an attention mechanism, was introduced by the research team at Google a few years ago. It not only has incredibly high performance but also united two major research fields of machine learning, computer vision, and natural language processing for the first time. However, behind its remarkable success in implementation, there are still potential issues that underlie it. This work is focused on the attention mechanism, for its dominance in the area predominantly affects how machine learning progresses in the future. This paper reviews the development of attention, and its functions, along with many recent models. Besides, this paper will discuss issues related to attention mechanisms and their future directions.