DocStringEval: Evaluating the Effectiveness of Language Models for Code Explanation Through DocString Generation

Large Language Models (LLMs) have demonstrated impressive capabilities in generating high-quality code. Furthermore, instruct-finetuned versions of LLMs have shown remarkable proficiency in generating question-answering responses that explain program functionality. This work aims to establish a benchmark for program explainability through DocString generation and improve upon initial results. We employ instruct-finetuned versions of 4 different code-trained models with 6 parameter settings, and 2 general language models to prompt and generate DocStrings for given Python program classes. The generated DocStrings are then evaluated against ground-truth DocStrings using three key metrics: accuracy via BERT score, clarity via Flesch reading ease score, and conciseness using compression ratio. Our evaluation encompasses multiple models with varying parameters and architectures. Our findings indicate that LLMs excel at generating DocStrings for a given class with remarkable proficiency, particularly when meaningful variable, class, and function names are used. Notably, techniques such as Chain of Thought (CoT) prompting can further enhance DocString quality. Interestingly, our results also reveal that increasing model parameters does not always yield higher-quality results, and NLP-variant models perform comparably to code-variant models on accuracy and ease of reading metrics. Additionally, we observe that LLMs' ability to explain programs decreases with longer program lengths. This study demonstrates the potential of LLMs in generating high-quality DocStrings for Python programs. Our results highlight the importance of careful model selection, prompting techniques, and evaluation metrics in achieving optimal performance.