APAM: Adaptive Pre-Training and Adaptive Meta Learning in Language Model for Noisy Labels and Long-Tailed Learning

Practical natural language processing (NLP) tasks often exhibit long-tailed distributions accompanied by noisy labels, posing significant challenges to the generalization and robustness of complex models such as Deep Neural Networks (DNNs). Traditional resampling techniques like oversampling or undersampling, while commonly employed, can easily lead to overfitting. A growing trend involves leveraging small amounts of metadata to learn data weights, alongside the demonstrated benefits of self-supervised pre-training, particularly for under-represented data. In this work, we propose a general framework that addresses both the long-tail and noisy label issues. Our model is adaptively tailored to the problem domain using a contrastive learning approach. The re-weighting module, a feed-forward network, learns explicit weighting functions and adjusts weights based on metadata. Additionally, our framework modifies the weights of terms in the loss function through a combination of polynomial expansion of the cross-entropy loss and focal loss. Extensive experiments consistently demonstrate the superior performance of our proposed framework compared to baseline methods. Finally, our comprehensive sensitivity analysis underscores the efficacy of the proposed framework in handling long-tailed problems and mitigating the adverse effects of noisy labels.