Where Does mBERT Understand Code-Mixing? Layer-Dependent Performance on Semantic Tasks

Semantic tasks like lexical relation prediction and word analogy are crucial for deep language understanding, yet pose significant challenges when applied to code-mixed text, where multiple languages are interwoven. This study investigates the ability of multilingual BERT (mBERT) to capture semantic nuances in code-mixed scenarios (English-Spanish and English-Telugu) through layer-wise probing across four distinct tasks: lexical relation prediction, word analogy prediction, sentiment analysis, and intent classification. We employ probing classifiers trained on representations extracted from each of mBERT’s 12 layers to assess semantic knowledge capture throughout the model. Our comprehensive experiments reveal that mid-level layers (approximately 4-7) generally exhibit the strongest performance for tasks like lexical relation and intent classification in code-mixed settings. However, performance varies significantly by task, with sentiment analysis proving particularly challenging (achieving near chance-level accuracy), highlighting mBERT’s limitations in capturing certain code-mixed semantic subtleties. This work contributes a multi-task layer-wise analysis of mBERT’s semantic capabilities on code-mixed data, providing insights into optimal layer usage and pinpointing areas requiring further research for robust multilingual, code-mixed natural language understanding.