Generative AI Efficiency and Effectiveness in Software Project Documentation Review Process

In this research, we report the capabilities of Large Language Model (LLM)-based Generative Artificial Intelligence (GenAI) tools, such as ChatGPT and Gemini, in the software project documentation review process. In software projects, the document artifact review process is an essential part of the quality assurance process. This review process is a human effort-intensive process. Introducing automation in this process will help reduce human effort, speed up the development, and reduce the potential of errors in the documents. With its enhanced natural language processing capabilities, LLM-based GenAI tools provide significant potential in the above process. In this research, we investigate the efficiency and effectiveness of LLM-based GenAI tools (such as ChatGPT and Gemini) for software documentation review process. Our experimental design includes comparing the manual document review process with the automated document review process conducted by LLM-based GenAI tools. The research results indicate that utilizing LLM-based GenAI tools provides significant efficiency in the document review process. However, LLM-based GenAI tools can only reach half the effectiveness and review quality that can be achieved by a human. To the best of our knowledge, this study is one of the first studies investigating the effectiveness and efficiency of utilizing GenAI for the software documentation review process. This research provides significant insights into improving the software document artifact review process utilizing LLM-based GenAI techniques.