Multilingual Natural Language Understanding for the FPT.AI Conversational Platform

This paper presents a multilingual natural language understanding model which is based on BERT and ELECTRA neural networks. The model is pre-trained and fine-tuned on large datasets of four languages: Indonesian, Malaysian, Japanese and Vietnamese. Our fine-tuning method uses an attentional recurrent neural network instead of the common fine-tuning with linear layers. The proposed model is evaluated on several standard benchmark datasets, including intent classification, named entity recognition and sentiment analysis. For Indonesian and Malaysian, our model achieves the same or higher results compared to the existing state-of-the-art IndoNLU and Bahasa ELECTRA models for these languages. For Japanese, our model achieves promising results on sentiment analysis and two-layer named entity recognition. For Vietnamese, our model improves the performance of two sequence labeling tasks including part-of-speech tagging and named entity recognition compared to the state-of-the-art results. The model has been deployed as a core component of the commercial FPT.AI conversational platform, effectively serving many clients in the Indonesian, Malaysian, Japanese and Vietnamese markets–the platform has served 62 million API requests in the first five months of 2022 for chatbot services.¹¹including requests deployed for on-premise contracts.