Assessing the Impact of ChatGPT in a PHP Programming Course

ChatGPT changed the way of learning for both instructors and students. Since its introduction, it has attracted a lot of attention from learners as well as instructors. In this paper, the impact of ChatGPT in a PHP programming course using user studies was investigated. User studies were conducted in two different universities in North Cyprus with a total of 50 students. Students were divided into two groups and asked to perform two quizzes; (a) manually alone and (b) with the assistance of ChatGPT. To remove the learning effect, quizzes were swapped; the first student performed the quiz manually first, then perform the second quiz with the help of ChatGPT with similar questions. Subsequently, the second student performed the quiz using ChatGPT first, then perform the second quiz manually next. Swapping continued for all students. Furthermore, to understand the impact of ChatGPT on different question types in a programming course, the quizzes were designed with different question categories: Classical, True/False, multiple choices, and coding. After completing each quiz (manual or assistance of ChatGPT), post-questionnaires were also given to assess the attitudes of learners to the exams. Results of the user study were analyzed in terms of scores (correct answers), post-questionnaires as user attitudes and statistical paired t-tests. Results indicated that ChatGPT had statistically significant positive effect on coding questions, as well as, statistically moderate positive effect on classical and True/False questions. However, for multiple choice questions, there is no significant difference between the results of manual exam and exam with the assistance of ChatGPT for the programming course. User ratings for post-questionnaires also confirm these results.