Research on Credit Scoring Based on Transformer-CatBoost Network Structure

Credit scoring models are widely used in banks and other financial institutions to make credit granting decisions. Over the past few years, many supervised machine learning algorithms have been successfully applied to credit scoring models. These models utilize consumers' historical data to assess the risk associated with an application. However, there are few consideration of integrating supervised learning with unsupervised learning. A hybrid model was proposed in this paper, combining the Transformer networks with the CatBoost decision tree. Firstly, the Transformer network encode and mine the internal information of the original data. Then the CatBoost model train the encoded vector to get the leaves nodes for generating cross features. The novelty of our approach relies on reducing the redundant and noisy features and avoiding the annotation for massive data. Experiments are carried on a real world dataset from a Chinese commercial bank, using to-fold cross validation. The results show that the AUC index and KS index of our model outperform the traditional machine learning methods and the single Transformer network, which confirm the superiority of the integration of unsupervised and supervised machine learning methods.