Enhancing Financial Sentiment Analysis Ability of Language Model via Targeted Numerical Change-Related Masking

Sentiment analysis is a critical task that is highly beneficial to various financial tasks such as stock-price prediction, corporate credit rating, economic report analysis, and investment decision support. Researchers have used various methods to train pretraining language models (PLMs) for these tasks. Although most PLMs have achieved excellent performance, they can be further improved. In this study, we propose a new framework to strengthen numerical understanding, in particular for the FinBERT(Financial Bidirectional Encoder Representations from Transformers) model released in 2019, thus improving model performance in the task of sentiment analysis on financial news sentences. This method selects sentences containing numerical words from financial news articles, preferentially masks the words, and post-train the PLM. To evaluate the proposed methodology quantitatively, we apply the same post-training to different financial language models and compare the performance before and after the application using Financial Phrasebank, which is a representative benchmark dataset used in financial sentiment analysis. The experimental results show that the best performance is achieved when 50,000 sentences are used to post-train FinBERT, thus confirming the advantage of the proposed methodology for downstream tasks and highlighting the importance of using the correct amount of data. Additionally, we show that applying the proposed method to different language models improves the performance, particularly in low-resource environments with less training data. The findings of this study suggest that the PLM can improve aspects that it does not understand well, and that thd PLM performance can be improved by post-training it with task- and domain-appropriate datasets, in not only finance but also in other domains.