Decoding Algorithmic Diversity Across Programming Languages using CodeT5+ with Clustering Analysis

Any coding question may have multiple possible correct answers. While assessing any such answer, an examiner evaluates the code sequence, logic and syntax to provide feedback or award marks. An automation system which is designed to assess such answers, depends on the systems ability to transform the unstructured code text to a vector space where these answers form coherent clusters. In this research, multiple correct answers for the same questions were analyzed through clustering analysis. Code snippets in Java, C and Python are converted to their vector embeddings using CodeT5+. Effectiveness of the clustering was considered in three different embedding scenarios: code only, Syntax Tree (AST) only, and code with AST. Various clustering algorithms included DBSCAN, HDBSCAN, Agglomerative Clustering, and Gaussian Mixture Models to group similar vectors. Additionally, the objective included to evaluate the dependency of coding language for effectiveness of cluster formation. The findings demonstrate that Code only embeddings delivered superior clustering where clusters formed are closer to human grouping. Additionally, the study demonstrates that the embeddings formed by CodeT5+ are independent of coding language. This suggests that the effectiveness of a system built for automatic grading of programming assignments may work equally well across a variety of programming languages.