Memory Capacity Assessment of Large Language Models: A Knowledge-Frequency Perspective

Memory capability is a critical aspect of large language models (LLMs). However, the disparity in memory ability between small and large LLMs remains unclear. In this paper, we present a novel investigation into the memory capabilities of both small and large LLMs, introducing an innovative knowledge-based dataset, enriched with knowledge frequency annotations. Derived from Wikidata, this dataset quantifies knowledge frequency by counting the co-occurrences of head and tail entities of knowledge graph triples in part of the pre-training corpus. Building upon this, we constructed a fact-based question-answering dataset called KDF, and evaluated the memory performance of state-of-the-art pre-trained base model families. Our comprehensive experiments demonstrate that large LLMs exhibit robust memory capabilities, retaining most facts even when they occur infrequently. Conversely, small LLMs are limited to recalling only a subset of high-frequency facts, struggling significantly with low-frequency information. Our study not only illuminates the memory discrepancies between different scales of LLMs but also offers a valuable resource and methodology for future research in LLMs.