Beyond the Hype: Perceptions and Realities of Using Large Language Models in Computer Science Education at an R1 University

With the mainstream adoption of Large Language Models (LLMs) over the last year, members of both academia and the media have raised concerns around the potential impact on student learning and pedagogy. Many students and educators wonder about the pedagogical fit of this emerging technology. We aim to measure the adoption and perception of LLMs among the CS education community in an R1 University to distinguish reality from hype. To this end, we conduct a large survey study targeting three populations participating in computing courses at the university: intro-sequence students (ISS), experienced students (ES), and faculty. Our survey seeks to gather insight around the different populations' perceptions of LLMs in education, as well as how these perceptions may be changing as LLMs improve. Our results show several significant differences across the views of 760 respondents. Most students report LLMs' un-paralleled potential for quick information access, yet many harbor concerns about their reliability and impact on academic integrity. Additionally, while ES rapidly integrate LLMs into their learning, ISS and faculty remain cautious, highlighting a stark contrast in adoption rates. Faculty are unconvinced of LLMs' educational benefits and are concerned about potential challenges in evaluating students' learning outcomes. LLMs are reshaping pedagogical approaches and student engagement. However, with the notable reservations expressed by certain segments, particularly by faculty and ISS, there is an imperative for careful, informed, and ethical integration to ensure that these tools enhance rather than compromise the educational experience.