Building a Language Model from Scratch

Summary

This chapter presents the evolution of language modeling using the transformer architecture to illustrate how machine learning models, specifically, large language models like LLaMA (Large Language Model Meta AI), came to be. The sinusoidal position information is hard‐coded. However, it is also possible to learn to represent the position information. An approach to learning to represent positional information is to treat positions as a categorical feature. Then, through back‐propagation, the network learns to efficiently encode positional information. Normalization can help stabilize neural network training by reducing the likelihood of exploding or vanishing gradients and improving model generalization. In the original transformer paper, the authors used position encodings added on top of the input embedding vectors of each word token to allow the attention model to disambiguate words from different positions along the sequence.