A Novel Skip Connection Structure in Transformer

Herein, a novel skip connection structure with double bypass is employed in Transformer. The structure introduces two linear components between the input and output. Moreover, it establishes layer normalization at the beginning of the bypass computation, enhancing the model prediction efficiency. Two methods are employed to deploy the novel skip connection structure in the Transformer. In one method, the Transformer feedforward layer used for machine translation tasks is considered as a branch of the skip connection structure. In the other method, the skip connection structure is introduced into the Transformer model used for binocular parallax calculation tasks. Experimental results show that the machine translation model with the novel skip connection structure is 2^% better in BLEU index. Furthermore, in the binocular parallax calculation tasks, the EPE index of the Transformer model with the skip connection structure decreases by 3.8% compared with that of a comparison model. The Transformer with the novel skip connection structure achieves a 7.6% lower 3PX ERROR than the Transformer without the skip connection structure.