From Code to Ratings: Converting Programming Data to Enhance Collaborative Filtering in Educational Online Judge Systems

This study introduces and compares three innovative approaches for recommending programming problems within an Online Judge system (OJ), tackling the challenge of deriving implicit ratings from user interactions without explicit user ratings. Conventional collaborative filtering (CF) methods often struggle with the sparse and implicit feedback typical in educational settings, limiting recommendation accuracy. A common method to handle implicit feedback is to use a binary system, which in the context of programming problems, simply records whether a user solved a problem (1) or did not (0). However, this method lacks scalability, fails to capture satisfaction levels, and overlooks problem difficulty and user skill. We analyzed submissions on the OJ to develop methods that generate implicit ratings reflecting user engagement and problem-solving performance. We proposed three distinct approaches, i.e., a Direct Weighted Sum Approach, a Formula-Based Approach and a Distance-Based Approach. Each approach determines user satisfaction, scaled from 1 to 5, and utilizes this in a Neural CF model to improve recommendation accuracy. Additionally, we included a baseline dataset that categorizes exercises in a binary format, marking them as either solved or not. The results demonstrate that all the approaches performed well in terms of accuracy. However, the proposed approaches significantly outperformed the baseline approach in recommendation quality by presenting relevant items to users earlier and offering a broader range of items. Another advantage of the proposed approaches is their consideration of problem difficulty and user skill, enabling a better understanding of how challenging each recommended item might be for each user.