How Big Can It Get? A comparative analysis of LLMs in architecture and scaling

Large Language models (LLMs) are increasingly becoming an integral part of our society. They play important roles in education, marketing and healthcare. These models exhibit emergent behavior when scaled, however such scaling might have its disadvantages as well. As a way to combat such disadvantages, new architectures and types of models have been designed. In this paper, we show the different architectural decisions when it comes to building LLMs as well as discuss how big a model needs to be in order to be specialized in a certain area or sector. We show that most specialized models are small in size yet outperform larger models in specific domain tasks.