A Multi-Label Classification Approach for Categorizing Beginner Programming Problems from Online Judges

This is a full paper, and it belongs to the research category. Online Judges (OJs) have been used by students and programming practitioners to solve problems by submitting solutions as source code and receiving immediate feedback. This important characteristic of OJs makes the learning and teaching of computer programming more productive and efficient, and they are considered valuable in several programming courses as well as by self-study students aiming to improve their programming skills. Moreover, OJs store a repository of problems that are mostly categorized according to topics related to competitive programming (such as graphs, dynamic programming, etc.), Although these problems are inappropriate for introductory programming courses, few OJs cover beginner-level programming problems categorized into basic topics, such as conditionals, loops, arrays, etc. Providing such categories is time-consuming and prone to error when done manually by faculties that formulate the proper categories and code solutions for all problems in the repository. This scenario motivated us to explore natural language processing (NLP) approaches to automatically classify beginner programming problems by analyzing their statements. For modeling this multi-label text classification approach, a corpus of beginner problems from a Brazilian OJ was manually curated using predefined class labels related to an introductory programming course. Experiments were performed using well-known classification models: Support Vector Machines (SVM), Gradient Boosting, Long Short-Term Memory (LSTM), Bidirectional LSTM, and BERT. The results showed that SVM yielded the best overall macro F1-Score, though class imbalance affected performance on individual topics.