Bag of Tricks for Citation Intent Classification via SciBERT

Citation intent classification task has a long history starting from manual and rule-based methods several decades ago. We follow its variant stated as multiclass classification of sentences with references. Last years with the rise of deep learning models and transformers many solutions of NLP tasks were improved and citation intent classification task was one of them. The big problem of the field is limited availability of labelled data. We focused on two recent and relatively big datasets: ACL-ARC (6 classes, 1941 instances) and SciCite (3 classes, 11020 instances) and tried to apply all modern techniques. It is not so easy to fine-tune transformers on small imbalanced datasets with a large number of classes. Therefore we proposed combination of tricks for SciBERT fine-tuning, which allow us to achieve 0.72 Macro-F1 on ACL-ARC dataset which is slightly better than previously reported results.