Evaluating the Effectiveness of Voice and Text Software Chatbots on Students’ Programming Skills

Artificial intelligence chatbots are increasingly used in programming education to provide real-time, personalized support. While text-based chatbots have been widely explored, research investigating how different communication modes—particularly voice versus text—affect programming learning, engagement, satisfaction, and code quality remains very limited, especially for Arabic-speaking students. This study compares the effectiveness of voice-based and text-based chatbots during a Java programming task. The experiment involved 30 second-year computer science students from Birzeit University who were all Arabic speakers. A hidden human (Wizard-of-Oz technique) simulated AI chatbot responses in Arabic during a one-hour task. Data were collected through system logs, surveys, interviews, and expert code reviews. Results showed that voice-based interaction led to higher engagement, improved code quality, and more positive experiences in terms of satisfaction and usability compared to text-based interaction. These findings support the development of voice-enabled, emotionally responsive, and personalized AI chatbots that better serve the needs of Arabic-speaking programming learners.