A Weighted Cross-entropy Loss for Mitigating LLM Hallucinations in Cross-lingual Continual Pretraining

Recently, due to the explosive advances of large language models (LLMs) on English, cross-lingual continual pretraining has been widely applied in obtaining Chinese LLMs. However, previous studies showed that these LLMs have suffered severe hallucinations, mainly caused by noisy tokens. To this aim, we propose a novel loss function, InfoLoss for continual pretraining. Specifically, our loss function takes into account the co-occurrence of noisy and normal tokens, and uses point-wise mutual information to reduce the impact of noisy tokens. We use InfoLoss to continually pretrain 30 billion tokens on Llama 2-7B with 64 A100 GPUs for 24 days, obtaining C-Llama. We then conduct experiments on 12 benchmarks for evaluations. The results show the effectiveness of our proposed InfoLoss. Our datasets and codes are publicly available at https://github.com/Fluxation996/C-Llama.