Computing Education Research Compiled: Keyword Trends, Building Blocks, Creators, and Dissemination

The need for organized computing education efforts dates back to the 1950s. Since then, computing education research (CER) has evolved and matured from its early initiatives and separation from mathematics education into a respectable research specialization of its own. In recent years, a number of meta-research papers, reviews, and scientometric studies have built overviews of CER from various perspectives. This paper continues that approach by offering new perspectives on the past and present state of CER: analyses of influential papers throughout the years, of the theoretical backgrounds of CER, of the institutions and authors who create CER, and finally of the top publication venues and their citation practices. The results reveal influential contributions from early curriculum guidelines to rigorous empirical research of today, the prominence of computer programming as a topic of research, evolving patterns of learning-theory usage, the dominance of high-income countries and a cluster of 52 elite institutions, and issues regarding citation practices within the central venues of dissemination.


I. INTRODUCTION
In the past 20-25 years, computing education research (CER) has matured from being an interest of teachers in colleges and universities, from all walks of computing research, into a respectable research specialization of its own [1], [2]. In the past two decades, the field has seen all the signs of an established field emerge: there are journals specialized in CER, research conferences, research centers, the field's own unique conceptual frameworks, seminal publications, and professorial appointments explicitly dedicated to the field [1]. But over the many decades that have passed, from CER's early beginnings as informal meetings to today's flourishing research area, the field has seen massive changes. Trends come and go, groups form and dissolve, expectations and concerns of computing change, and technologies rise and fall with the hype cycle. In order to understand the present-day issues of CER and to appreciate the field's current debates, it is important to recognize its history and evolution. By building a narrative of the past, it is possible to see relationships between societies and people; to build a sense of identity; and to weave a narrative of authors, institutions, and their networks, as well as where communities have come from, and potentially reveal clues as to where the field of CER is headed.
This article complements previous reviews, overviews, and meta-research of CER by offering four unique scientometric perspectives of the past and present of CER. First, the article provides a historical overview of the influential publications and dominant topics and themes of research over the past six decades of CER. Second, it investigates the co-citation metrics of CER publications in order to identify the common building blocks of CER and gain insights into the foundational studies that CER builds upon. Third, it analyzes the creators of CER: influential authors, institutions, and networks of collaboration. Finally, because a crucial part of science is dissemination, where the central venues act as gatekeepers to control what is valued and accepted and what is not, the article identifies the most influential publication VOLUME 4, 2016 venues of CER and analyzes their citation practices. The study relies on Scopus metadata from CER publications imported from the most influential dedicated publication venues of CER, as well as CER publications identified by a common keyword search. The article concludes with a discussion and reflection on the findings.

A. A BRIEF HISTORY OF CER
There is not a single birth date for computing as a discipline, as it gradually separated itself from a number of existing academic fields [3]. Similarly, there is no single moment when computing education separated from courses on numerical mathematics, electrical engineering, and natural sciences, among others [4], [5]. The mass production of stored-program computers required large numbers of programmers and, as early as 1954, a conference on computing education acknowledged the need for organized computing education efforts [6]. By the early 1960s, organizations like DPMA (Data Processing Management Association) and IFIP (International Federation for Information Processing) presented their own curricula, private companies' own training programs flourished, and computing education had become institutionalized in a large number of universities [7].
As computing programs mushroomed in colleges and universities through the 1960s, increasing numbers of computing educators wished to advance the art of computing education. Conferences were an important venue for airing ideas about computing education through the 1960s: among numerous others, IFIP's TC-3 (established in 1963) and Association for Computing Machinery (ACM)'s SIGCSE (Special Interest Group in Computer Science Education, established in 1968) both organized their first conferences in 1970. Empirical results were published in journals like International Journal of Man-Machine Studies, the Computer Journal, Computers & Education, and Communications of the ACM. Along with countless course descriptions, new learning tools, experience reports, and ideas for how to best teach computing topics, there were also scattered examples of empirical research on computing education [8], [9]. From the 1980s and 1990s on, there was a growing interest in studies on learning programming languages, on how learning programming teaches "generic" skills, and on the psychology of programming [e.g., the Psychology of Programming Interest Group (PPIG) and the Empirical Studies of Programmers (ESP)], to mention a few [9], [10]. At the same time there was a growing feeling that "swap meets" for sharing one's pedagogical ideas, useful assignments, or lecture props [11] do not suffice for building a respectable body of knowledge on computing education.
A number of shifts started around the year 2000. From the early 2000s on, computing symposia started to place increasingly stringent restrictions on the methodological quality of the papers they published [1], [12], several capacitybuilding initiatives aimed at training computing education researchers for disciplinary based education research [13], and a few landmark works re-defined the computing ed-ucation research landscape (e.g., [14]). As ACM's prime journal on computing education was re-branded from JERIC (Journal of Educational Resources in Computing) to TOCE (Transactions on Computing Education), it announced a shift away from engineering/technology articles [1, pp. 64, 80], and new conferences like ICER (the International Computing Education Research Conference) started to expect methodologically rigorous, theory-informed studies.

B. RELATED RESEARCH
Over the years, the CER field has seen a number of analyses, reviews, and meta-research on CER publications. One of the earliest was that of Valentine [15], who classified SIGCSE Technical Symposium papers as Marco Polo, Tools, Experimental, Nifty, Philosophy, and John Henry. Valentine's research soon sparked a number of other efforts to analyze CER papers (e.g., [11], [14], [16]- [18]). Meta-analyses of papers published in single CER venues include analyses of the ICER (International Computing Education Research) conference [19], [20], the ACE (Australian Computing Education) conference [17], [21], the ITiCSE (Innovation and Technology in Computer Science Education) conference [22], [23], Koli Calling conference [24], [25], a comparison of research in Koli Calling and Informatics in Education journal [26] and the NACCQ (National Advisory Committee on Computing Qualifications) conference [27]. Several analyses and reviews have focused on papers on computer programming [8], [28]- [30], and in K-12 education [31], [32]. One of the most prolific analysts, Simon, developed a classification system to categorize CER into four dimensions: context (the subject matter), theme, scope, and nature (an experience report, empirical analysis or experiment) [1], [17]. For instance, a recent analysis showed that today's ACE conference has more research papers than experience reports-especially research studies on computer programming [21]. The extensive work of Simon is well summarized in [1].
Reviews focusing on theory use in CER have started to appear. A study [33] reviewed 72 papers from ICER (2005)(2006)(2007)(2008)(2009)(2010), and found that 57% of papers used a theoretical basis (a theory, model, or framework), with the most common theories being Bloom's taxonomy, cognitive apprenticeship, cognitive load theory, schemata theory, self-efficacy theory, situated learning, structure of the observed learning outcome (SOLO) taxonomy, and threshold concepts [33]. Researchers went on to analyze TOCE and Computer Science Education (CSE) publications (n = 308) [34], and found that 51% of papers had a theoretical base, mostly drawn from computing, psychology, and education. In recent work [35], the target was theoretical constructs (TCs) with a specific origin in CER, with findings showing common examples of TC's such as proximal flow, learning edge momentum, and others, but still an overall modest use of TCs with no prevailing theoretical works that are broadly applied [35]. The results do point towards gradual maturation of CER through building its own theoretical constructs and in this way claiming its independence [35]. A recent review on learning theories in CER [36] identified a list of common learning theories and related influential contributions. The use of theories in CER articles in the central venues of TOCE, CSE, SIGCSE Bulletin, SIGCSE Technical Symposium (TS), ICER, ITiCSE, Koli Calling, and ACE were inspected. Top referenced learning theories were identified, including flow-theory, self-efficacy, and the largely debunked learning styles theory [36]. The findings also show that a large majority of influential papers were found to have only a limited reach within CER [36].
Use of research design and research methodology has been the target of many analyses. For example, a recent review of 427 papers in 2014 and 2015, published in SIGCSE TS, ICER, ITiCSE, TOCE, and CSE, found that 80% of papers included some form of empirical evaluation, and that many papers lacked rigor in reporting and did not consistently follow reporting norms [37]. While empirical research is on the increase, it still lacks in reporting quality. In recent years, venues such as SIGCSE TS, and TOCE have started to pay increased attention into reporting of empirical work in their review criteria. It has been estimated that this will eventually lead to better support for replication and meta-analyses in the future [37]. Other reviews to investigate methodology use in CER include those of [28], [33], [38], and findings show, e.g., extensive use of statistical methods beyond descriptives [28], and reveal what statistical methods are commonly used in CER [38]. A review [39] reported that 54% of empirical papers in SIGCSE TS (2014-2015) used surveys for data collection, and a small percentage reported use of qualitative methods, while another review [40] showed that 32% of papers in TOCE, CSE, and ICER (2013-2017) used both qualitative and quantitative data. Other reviews have looked, e.g., at the replicability of CER studies [41].
Scientometric analyses of CER publications are also starting to emerge. One study analyzed the collaboration networks of SIGCSE, ITiCSE, and ICER, revealing the dominance of high-income countries, especially the United States, and that, while collaboration between authors from different institutions is growing, it still mostly happens within the same country [42]. The geographical diversity in the SIGCSE Technical Symposium has been investigated, with results showing strong US dominance [43], [44]. One scientometric study investigated the publications in the ASEE/IEEE Frontiers in Education (FIE) conference from the viewpoint of most cited papers, influential authors, and internalization [45], while another analysis of FIE looked at sources of funding [46]. Another piece of research scientometrically analyzed the keywords in ITiCSE and ICER papers [47], showing, for instance, a steady trend of research on introductory programming courses. The history of computational thinking (CT), a subarea of CER research, was scientometrically mapped by a recent study [48].
The combined body of literature reviews, meta-research, and scientometric analyses of CER paints a picture of CER evolving over the decades from experience reports to empirical research; increased attention paid to learning theories, research methods, and reporting rigor; sustained focus on programming education; and dominance of high-income countries, especially the US. Up-to-date, scientometric studies have targeted one or two publication venues, missing a holistic and more in-depth analysis of the CER field. No previous scientometric or other analyses have covered all central CER publication venues, over many decades, and offered a holistic analysis of citations, keyword trends, foundational work, dissemination, and core institutions that create CER. While a number of reviews and meta-reviews exist, their scope is limited to a restricted set of articles within a specified topic. All these lacks in the previous analyses together constitute the research gap for this paper.

C. SCIENTOMETRICS
The increased quality in structured databases such as Scopus and Web of Science, combined with new methods of data science and network analysis have resulted in rising popularity of scientometric studies [48], [49]. With new tools and methods, one can go far beyond simple quantities and descriptives. The new tools provide researchers means to conduct transparent and reproducible studies of published research. In this research, we apply state-of-the-art methods to offer quantitative in-depth views to CER. The tools used include the Bibliometrix package [50] for analysis, OpenRefine 1 for data screening, and Gephi 2 for visualisations.

D. RESEARCH QUESTIONS
As CER as a research field has evolved and matured over the years, its history offers a unique opportunity to study the evolution of central trends and themes in research topics (what CER studies), foundational work (what the studies are based on), knowledge creators (who does CER), and finally the dissemination (how CER is published). The research questions for this paper read as follows: 1) How has the publication profile of CER evolved in terms of keyword trends and topics of research? (RQ 1 ) 2) What do citation and co-citation metrics reveal about the foundational work in the CER discipline? (RQ 2 ) 3) How have the knowledge creators (authors and institutions) and their collaboration shaped the discipline of CER and its communities? (RQ 3 ) 4) How can the central venues of dissemination of CER be characterized with regards to their publication profiles and citation practices? (RQ 4 ) The paper is structured as follows: data and methods will be presented first in section II, followed by evolution of keyword trends in section III, foundational work (section IV), creators (section V), and finally dissemination in section VI. The results will be discussed in section VII. Finally, the article is concluded in section VIII.   [51]. Compared to the other databases, Scopus is well maintained, and it uses rigorous selection criteria for inclusion of journals and conference proceedings [52], [53].

B. DEDICATED VENUES SEARCH
All publications from well-known CER venues were included in the data extraction. These included the two major journals dedicated to CER, which are All papers published in the aforementioned conferences were included in the sample. For those conference proceedings that were missing some of their early years in the Scopus database (e.g., Koli Calling and the ACM SIGCSE Symposium), the missing data were manually retrieved from the ACM Digital Library and conference archives, where available.

C. KEYWORD SEARCH
A line needed to be drawn with regards to the amount of noise (non-CER articles) brought by less CER-focused publishing venues. For that reason some important venues were excluded. Most importantly, especially in the early days of computing education, CER results have been published in venues such as engineering education journals, generic computing journals, educational technology journals, and educational journals [7]. For example, the IEEE Symposium on Visual Languages and Human-Centric Computing, started in 1984 often included computing education research results [55]. Other venues known to occasionally publish important CER contributions-among other topics-include IEEE Transactions on Education, Communications of the ACM, IEEE/ASEE Frontiers in Education Conference, Informatics in Education, LATiCE (Learning And Teaching in Computing and Engineering, as well as the Psychology of Programming Interest Group (PPIG), formed in 1987, and the Empirical Studies of Programmers (ESP), with its first workshop in 1986 [55]. Due to the large numbers of non-CER articles, those venues were excluded from full inclusion, but in order to still capture a representative sample of CER from outside the realm of the dedicated venues, a search query was performed in Scopus to find additional relevant research in computing education that has been published in venues that are not mainly devoted to this field. To identify the best suited search terms, several iterations of search queries were conducted, followed by examining the resulting papers for relevance to the study. The following search terms were used. would amplify articles included in these reviews. CER has evolved from tools-research and experience reports to empirical research with increased attention in research methods and theories. As experience reports and tools-research have always been important parts of CER, and still are, we kept all such articles in the dataset. Articles published during 2021 were excluded to allow comparison across complete years.
The keyword "informatics education" is commonly used in German-speaking communities. However, a separate keyword search in Scopus resulted in several thousand contributions, many in the health informatics domain. By excluding health-related publication venues, and the articles already present in our dataset, the result set was limited to some 187 articles. A closer inspection revealed that those articles represent a diverse group of research areas, such as education technology and information technology (IT). A decision was made to exclude these articles, since many of them did not belong to CER.
After removing duplicates and uneligible articles, the number of resulting articles from combining both queries (full venues and keyword search) was 16,453 (see Fig. 1).

E. DATA SCREENING
Data screening means inspecting the data for errors and inconsistencies and fixing these errors in order to maximise the signal and minimise noise to prepare the data for analysis. For data screening, four steps were performed: 1) Author names were cleaned based on Scopus ID, so authors with different name spellings were combined. 2) Conference proceedings titles were checked and different editions of the same conference were combined (e.g., "SIGCSE 2018 Proceedings" and "Proceedings of the 2019 ACM SIGCSE Symposium"). Variations of the title of the proceedings for the same conference were also combined (e.g., "SIGCSE" and "SIGCSE Symposium"). Conferences with name changes were also combined. Similarly, variations in journal names were cleaned (e.g., "Computers and Education" and "Computers & Education"). 3) Institutions were manually cleaned and variations were combined (e.g., "University of Helsinki", "Helsinki University" and "The University of Helsinki"). 4) Keywords with similar meaning were combined including singular and plural forms (e.g., network and networks); abbreviations with their full spellings (e.g., "artificial intelligence" and "AI"), and variations of the same keyword, e.g. ("introductory courses", "introduction to CS", "CS1/CS2"), ("MOOC", "MOOCs", "massive open online course" and "massive open online courses"). The OpenRefine tool by Google was used to facilitate the process. The screening process and results are outlined in Fig. 1.

F. ANALYSIS: METHODS AND TOOLS
The screened data set was analyzed using the R statistical language with the Bibliometrix package [50]. The frequen-  cies of keywords, articles, citations, and plots were calculated from the Scopus metadata. A network of institutions was constructed from the affiliations of the co-authors of each article. Two institutions were considered connected if authors from both institutions collaborated on the same article. The network was plotted using the open-source software application Gephi [56] with the Fruchterman-Reingold layout algorithm. To study the presence of the rich-club phenomenon, the rich club coefficient (Ø) was computed as the ratio of the existing number of connections between the highly connected core (rich club) and the maximum connections possible. The boundary of the rich club core was computed using the method of decreasing rank order [57]. To exclude the possibility that the high degree of connectivity among the connected core nodes was a result of random chance, the normalized rich-club coefficient (Ønorm) was also computed. The Ønorm is the ratio of relative value of Ø to the mean value of Ø computed in 10 000 random networks that were matched for size and degree distribution.
A co-citation network was constructed by considering two papers connected when they were cited by the same paper. The network size was limited to the top 100 most referenced papers. To map the subgroup of references that are commonly cited together-representing themes of theoretical or foundational articles-community detection was done using Louvain modularity, a value of 1 was set for the modularity, and for the community detection, edge weights were prioritised [58]. The network was plotted using the Fruchterman Reingold layout algorithm and communities were color-coded. The node size is proportional to the degree centrality, i.e., the number of references connected to each node. A keyword co-occurrence network was constructed by considering each pair of two author keywords that appear in the same manuscript as connected. Groups or communities of keywords that are frequently listed together or strongly connected represent research themes. Louvain modularity was VOLUME 4, 2016 5 applied for community detection, with same parameters as in the detection of foundational articles. The keyword network was plotted in the same way as the co-citation network.

III. EVOLUTION OF TRENDS AND THEMES
In order to gain a picture of the evolution of trends and themes of research, we briefly discuss the evolution of CER through the decades (III-A), followed by an analysis of keyword trends in the past two decades, the time period in which keywords have been available (III-B), and concluding with an analysis of the most influential topics of research, as revealed by keyword clusters (III-C). Table 2 presents influential (highly cited) articles in the data over the decades of CER. Prior to 1970, academic publications on CER were scarce. Since the 1950s, universities have considered how to arrange computing education. In 1951, some early textbooks on programming were launched; in 1960s, the software industry grew, and universities started to offer training in computing and form computer science departments. In 1960, ACM started their education committee, and, in 1968, the ACM Computing Curriculum (CC) '68 [59] was published. The CC'68 became an authoritative guideline for establishing computing education in the US. Reviews of curricula, such as that of Stanford, and recommendations for establishing computer science departments were published [60]. A survey covering 25 US CS departments with students graduating in 1966-1967 revealed diversity in programs and, e.g., the wide teaching of FORTRAN, ALGOL, and MAD and the presence of AI in many curricula [61]. Another survey of computing curricula found that most computer professionals back then had reached their positions through apprenticeship or self-training [63]. The influential articles prior to 1970 are curriculum guidelines [62], recommendations, surveys, or reviews, which contributed to building the basis for teaching the then-new discipline of computer science. Publication venues for CER at this time were rare, and the many contributions consist mainly of e.g. curriculum guidelines rather than research articles.

A. SIX DECADES OF CER
The 1970s are marked by the development of communities of practice in CER, along with the formation of first conferences, journals, and magazines. The new computing curriculum CC'78 [64] introduced a shift from a mathematically oriented view of computer science (CS) to a more diverse discipline, including hands-on work, programming, and applications [64]. Influential work includes Marvin Minsky's Turing Award lecture [65] and recommendations for colleges, such as guidance for CS in small colleges [68]. Work on tools and educational technologies started, such as a report on an educational technology for learning programming: a diagnostic compiler that allows preparing, debugging, and executing simple programs [66]. A survey of CER literature post ACM's 1968 curriculum recommendations shows that CER to that date included research on tools and pedagogical aids, activity reports, and course descriptions, but only a limited number of empirical research papers [67]. Central to this decade were debates on programming languages and how to teach programming, theory versus practice, and the role of mathematics versus demands of the software industry. New discussion threads included those on programming teamwork, human-computer interaction, and professional accreditation [7].
In the 1980s, the debates on programming language and debates on how to teach programming continued [69]. Influential contributions included new curriculum recommendations [71], research on tools, and programming environments, such as the "TRY system" for program testing [72]. The decade was also marked with a growing number of empirical research, such as those of predicting success of freshmen [70], and predicting performance on introductory programming courses [73]. Although empirical research was starting to appear, it still took more than 20 years for CER to mature as a primarily empirical research field [1]. In 1989, Denning's influential article [90] was published, coining the term "computing", and made a remarkable impact on the 1991 Computing Curriculum. 1980s also marked the maturing of Human Computer Interaction (HCI) and usability into research fields, Papert's radical ideas, and demands for more experimental computer science [7].
Central characteristics of the 1990s are the "infocomputational turn", which started already in the 1980s; scientists from other fields started looking at their fields as information processes; problem solving becoming a central concept in CER; and diversification of academic contexts, such as those recommended by Computing Curricula 1991 [7]. Also, discussions on pedagogical aspects increased, as is demonstrated by a reflection on constructivism in CER [74], [75], and a paper on active learning [78]. Programming continued to be a central topic of research [76], [79]. One influential paper of the decade is a survey about the Entity-Relationship (ER) models [77], an important topic in software engineering.
In 2000s, the new ACM/IEEE computing curricula split computing into five disciplines of computer engineering, computer science, software engineering, information technology, and information systems, making it no longer a one-size fits all [7]. In addition, CER became established as a research discipline with new venues of dissemination established, and publication of the influential book of Fincher in 2004 [14]. Influential papers of the 2000s include a study on game-based learning in K-12 [80], and several studies on programming: difficulties of novice programmers [81], a multi-national study of assessment of programming skills [82], and an article about the role of Arduino and e-textile design in computing education [84]. Algorithm visualisation [83] attracted a lot of interest. A large amount of research was on computer programming. An increase in empirical research papers was seen, as CER continued to build its identity as a respected academic field [1].
From the 2010s onwards, CER became more established as a research discipline with formation and maturing of publication venues, establishment of professorships in CER, and a steep increase in the number of publications [1]. The highly cited papers include a paper on the Scratch programming language [85], a study on block-programming in K-12 [88], a paper about failure rates in programming courses [86], a visualization tool for Python [87], and a paper about measuring computational thinking skills in the K-12 domain [89].

B. ANALYSIS OF KEYWORDS
A unique view to influential topics, themes, and trends in CER during various time periods is revealed by analysis of keywords. The use of keywords prior to 2000 was rare and inconsistent, and while some publications prior to 2000 listed "themes" of research, those themes are not suitable for analysis. Therefore, reliable analysis of keywords is possible only post 2000. We have plotted the frequencies of top keywords between 2000-2020 ( Fig. 2a), ranks of top keywords between 2000-2020 ( Fig. 2b), proportions of top keywords between 2000-2020 ( Fig. 3a) and proportions of top keywords between 2011-2020 (Fig. 3b). Analysis of the top 20 keywords reveal several things. First, as observed in Sec. III-A, teaching and learning of programming has always been a dominant theme in CER. Our keyword analysis confirms, loud and clear, that teaching and learning programming continues to be a dominating area of research in the 2000s and 2010s, too. The frequencies of keywords CS1/CS2, novice programmer (referring to introductory courses in programming) and programming have all increased during the past two decades (Fig. 2a), and the ranks (Fig. 2b) show that the keyword CS1/CS2, which refers to research on introductory programming courses, has the highest overall rankings among all top 20 keywords, while programming, referring more generally to teaching and learning programming, also has a high ranking. The number of citations of papers with programming-related keywords is also remarkable. As revealed in Sec. III-A, and confirmed in the analysis of keywords, the roots of programming education as a central topic in CER are deep, and this trend continues in the 2000s and 2010s. The strong emphasis on programming is clearly visible in previous analyses of publications in many central venues of CER, too [1], [22], [45]. During all decades of CER, the choice of programming language has been a central concern and has sparked many heated discussions and debates [7]. Our keyword analysis shows that the keyword Java has experienced a downward trend, and no longer is among the top 20 keywords post 2010 (Fig. 3b).
Second, two top keywords with a remarkable and steep increasing trend over the past two decades are K-12 and computational thinking (CT). Starting from 2006, when also Jeanette Wing's seminal discussion on computational thinking [91] was published, the rankings of computational thinking and K-12 skyrocketed (Fig. 2b), which is also reflected in the remarkable increase in proportions of these keywords both post 2000, (Fig. 3a), and in the 2010s (Fig. 3b),  when these keywords reached the highest proportions within all top keywords, together with CS1/CS2, as confirmed by previous meta-analyses [48]. Third, the keyword LA-EDM (learning analytics and educational data mining) experienced a steep increase in popularity from 2012 onwards, reaching the top 20 keywords post 2000, despite being rarely seen as a keyword prior to 2010. The explanation is that 2010 was the time when a steep increase in overall research on learning analytics and educational data mining begun [92], and CER followed along with the trend. The beginning of 2010 sparked a steep increase in articles, published books, and new events on learning analytics and educational data mining such as the annual conference on Learning Analytics and Knowledge (LAK), which was launched in 2011 [92].
Fourth, it is a pleasure to see an increasing trend in research on gender and diversity. The keywords assessment and automatic assessment are also among the top 20, assessment experiencing a turbulent trajectory over the top rankings (Fig.  2b). In addition to exceptionally popular keywords such as CS1/CS2 and computational thinking, the top 20 keywords include pedagogical keywords, such as active learning, and collaborative learning, motivation, teaching, and pedagogy, which show that pedagogical considerations are important, or at least that they are often inserted as keywords in articles. Since the emergence and growth of the software industry in the 1960s, software engineering has always been a crucial part of CER [7], [64], and much continues to be so in the 2000s and 2010s, while educational technologies, elearning, game-based learning, and visualization are also included in the top 20 topics of research in CER.

C. KEYWORD CLUSTERS
In order to gain an alternative and complementing view about the central thematic areas of CER, we conducted an analysis of frequently co-occurring keywords. The network of cooccurrence was constructed on the basis of keyword communities identified by the Louvain modularity algorithm. Fig.  4 visualises the clusters of interconnected keywords, which represent the major themes of research within CER, post 2000. The analysis identified a total of nine (9) clusters. The four larger and dominating clusters are the orange, red, pink, and green clusters. Smaller clusters consist of blue, lightblue, light-magenta, purple, and gold clusters. While the nodes in the clusters are interconnected, so are the clusters. The presence of two large and interconnected red and orange clusters that center around programming-related topics confirms the dominance of research on teaching and learning to program in CER. While the orange cluster centers around first courses (CS1/CS2), object-orientation (OOP), Java, and gamification, the red cluster focuses more on teaching aids and tools, such as: e-learning, educational technology, and automatic assessment. The large green cluster confirms the rising popularity of K-12 and computational thinking, and complements the findings with keywords that are typically found together: gender and diversity is highly visible in this VOLUME 4, 2016 FIGURE 4: Associations between keywords, with keywords "computer science education" and "computing education" removed. The circle size represents count of keywords, edge thickness denotes frequency of co-occurrence, and colors indicate clusters of keywords.
cluster, together with broadening participation, and common tools in the K-12 domain: Scratch and robotics. Finally, the large pink cluster includes topics centered around curriculum, focused on software engineering, information systems, projects, and project-based learning, representing topics typically associated with software engineering. The smaller clusters center around game based learning (light-magenta), collaborative learning, active learning and pair programming (gold), pedagogy (blue), visualisation, algorithm design and data structures (light-blue), and evaluation and assessment (purple). Fig. 5 shows a network of the most co-cited papers, and we interpret these frequently co-cited constellations of keywords to be foundational articles: the building blocks of CER. These papers are not necessarily papers in our dataset, but contain papers outside of CER, e.g., learning theoretical contributions, and influential reviews of CER. They are the most cocited papers by the papers in our dataset. The papers are listed in Appendix A. The building blocks are organized in clusters based on the themes of research. Two separate clusters form the building blocks of programming research, with the first cluster having a basis more on general educational theories, while the second cluster bases its research more on CERoriginated theories and pedagogies. Another large cluster consists of the foundational papers on research about K-12 and computational thinking. Smaller clusters make the foundations on educational issues (gray), automated feedback and assessment (light green), meta-analyses (blue), and emotional aspects in programming (brown). It must be noted that we are interested in what CER research co-cites, regardless of the domain of the cited articles. By doing this we are able to investigate to what extent contributions of CER cite works from its own domain (CER), and on the other hand, from other domains, such as the learning sciences, mathematics education, or methodological literature. We follow approaches from previous co-citation analyses [93], [94].

A. GREEN CLUSTER: THE TRADITIONAL TRACK OF PROGRAMMING RESEARCH
The green cluster includes 26 papers, which are likely to be used as a frame in a lot of the research on learning and teaching computer programming. The top three papers in this cluster are a review of learning and teaching programming [10], an influential reflection on how introductory programming should be taught through algorithm development [69], and an influential paper about the role of constructivism in computer science education [95], where Ben-Ari discusses the importance of distinguishing constructivism in computing education from constructivism in natural sciences. The other papers include influential reviews on introductory programming [8], reviews on program visualizations, papers on exam questions, learning objectives, and articles about the relationship between reading, tracing, and writing in programming. Besides actual studies on computer programming, the foundational papers include five major theoretical foundations of learning, including the SOLO taxonomy [96], cognitive load in problem solving [97], Bloom's taxonomy [98], revised Bloom's taxonomy, and constructivism in computing education [95], with two additional papers discussing the role of SOLO taxonomy in programming. Previous work has identified Bloom's taxonomy, cognitive load theory, SOLO taxonomy, and self-efficacy theories as popular theories in CER [33], [36]. Out of the 27 foundational articles in this cluster, with the exception of the learning theoretical contributions, many papers are from within the CER domain: influential reviews, meta-reviews or large-scale questionnaires on teaching and learning programming. A few papers are individual influential papers on visualization tools. There are no papers on research methods or research design in this cluster.

B. ORANGE CLUSTER: A MODERN TRACK OF PROGRAMMING RESEARCH
The orange cluster includes 26 papers, which are also likely to be used as a foundation in papers that deal with research on teaching and learning to program. The top three papers are a survey among institutions around the world about failure rates in CS1 courses [99], a study of compilation behaviour: approaches that students take when they engage in repeated editing and compiling of their practice programs during learning [100], and a study of success factors in an introductory computer science course [101]. A large share of the other papers are studies on failure rates, success factors, predicting performance, or understanding reasons why students drop out of programming courses. A number of new pedagogical approaches appear, including the "extreme apprenticeship" method [102], and several papers on pair programming. As opposed to the green cluster, the orange cluster contains no learning theoretical constructs outside CER but includes pedagogies designed and contextualized specifically for learning and teaching programming by CER researchers. While the green cluster leans towards common learning theories, the orange cluster leans on pedagogies developed inside CER and therefore represents a new wave of research on developing discipline-specific theories, constructs, and pedagogies. The orange cluster does not include any papers on research methods or research design.

C. PINK CLUSTER: CT AND K-12
The pink cluster consists of 23 papers, which form the foundation in a major share of papers that deal with research on the K-12 and computational thinking track. The most influential paper in this cluster is Jeanette Wing's discussion paper from 2006 [91], which marked the time when the term "computational thinking" entered the common computing education vocabulary. The cluster includes Papert's seminal book on Mindstorms [103], and a taxonomy of programming environments [104]. In addition, a number of reviews on CT VOLUME 4, 2016 The cluster includes a number of foundational papers on educational technologies and pedagogies; LEGO Mindstorms, Scratch, Alice, and Computer Science Unplugged. This cluster contains one methodological contribution: Cohen's book on statistical power analysis [105], introducing the Cohen's d for estimating sample sizes and evaluating strengths of statistical claims. This suggests that well-known papers in the CT and K-12 track have used Cohen's methods, e.g. the effect size coefficient in their research on testing pedagogies, educational technologies or CT interventions and their relationship to, e.g., learning outcomes or motivation. These articles are co-cited by a major share of CER in the CT and K-12 domain. There are many recognised contributions to CT (e.g. [106]- [109]), but not all of them were found in the top co-cited articles in our analysis.

D. THE GRAY CLUSTER: FOUNDATIONS ON SOCIAL ASPECTS AND DIVERSITY
The gray cluster covers the foundations of research on social aspects and diversity, including a book on the gender-gap in computing [110], an influential book on situated learning, the social aspects of learning, stressing the importance of communities of practice, and knowledge creation in communities [111]. The cluster involves grand works on education including that of Vygotsky [112], Bandura's self-efficacy [113], and constructivism. Self-efficacy, constructivism, and communities of practice were identified as top referenced learning theories in a recent review of CER, dispite having a relatively limited overall reach within CER [36]. The cluster also includes articles about gender differences, and mental models in programming. The cluster represents the building blocks of work in CER research that deals with social aspects, diversity, communities of practice, and knowledge building in communities.

E. SMALL CLUSTERS: AUTOMATED FEEDBACK, META-ANALYSES, EMOTIONS
The light green cluster includes four influential papers on automated feedback and assessment in programming, forming the foundations for research on automatic assessment. The blue cluster is based on a seminal meta-analysis of SIGCSE proceedings [15], acting as a building block for reviews and meta-analyses of CER. The brown cluster includes a lone paper on the emotional toll of programming assignments in introductory programming, acting as a building block for research on affective aspects in programming.

V. THE KNOWLEDGE CREATORS OF CER
In this section we look into the knowledge creators of CER by zooming in on influential institutions and authors, and their collaboration networks. First, we conduct a "rich-club" analysis of CER institutions (V-A), followed by a peek into the most influential authors (V-B) and geographical distribution of CER (V-C).

A. A DOMINATING RICH CLUB OF 52 INSTITUTIONS
Our data contains some 4011 unique affiliations (institutions) that have produced CER. The functioning of collaborative groups is the backbone of efficient co-creation of knowledge [114]. In a network of collaborators, the concept of rich club refers to a situation in the network, where highly-connected nodes or hubs interact primarily among themselves, indicating a dominance, or oligarchy, of teamwork in a way, which may make the network less collaborative as a whole. In other words, a rich club is a small but dominating subset in a network [115]- [117]. In learning contexts, students who are left outside of a "rich club", may become isolated, lose motivation, or become underachievers [118]. It is not uncommon for high achieving students to build a rich club, excluding lower achievers from information exchange [118]. All forms of computing are done within networks of people, and the history of computing is also a history of discrimination, biases, well-functioning collaborative groups, and various forms of rich clubs. In this case, we make an investigation into institutional-level rich clubs within the discipline of CER.
Our analysis shows that out of all the 4011 unique affiliations (institutions) in the data, some 1.3% (n = 52) belong to a rich club (see Fig. 6). The 52 institutions in the rich club (shown with green color in Fig 6, and listed in Appendix B) were involved in some 9% of all articles in the dataset. Those  Fig. 7 shows the top 20 influential authors (in number of contributions in our data) and their production over time. The list is not intended as a ranking list, but rather as a sample of some of the giants of CER: authors who push the boundaries of CER. Many other influential authors exist. There is an equal balance of 10 females and 10 males within these 20 influential authors.

B. INFLUENTIAL AUTHORS
The sample of 20 influential authors include Mark Guzdial, a multiple-award-winning author and inventor of the media computation approach to learning introductory computing; Simon, whose research focuses on programming education, and who is also known for inventing the Simon's system for classifying CER; Judithe Sheard, a long-time and awarded CER researcher, focusing particularly in the areas of educational technology, student learning behavior, and assessment; Beth Simon, with her research focusing on online and technology-enhanced learning and curriculum development for K-12; Robert McCartney, an awarded CER researcher and former editor of the ACM Transactions on Computing Education; Stephen H. Edwards, focusing on various CER projects in automatic assessment, metrics, gamification, and innovative teaching methods; Raymond Lister, a computer scientist and CER researcher specialising in understanding the learning and teaching of com-VOLUME 4, 2016 FIGURE 8: Global Distribution of CER puter programming; John Impagliazzo, author of numerous books, articles, contributor in computing education, computing history, and computing accreditation and outcome assessment, and receiver of many IEEE and ACM/SIGCSE awards; Andrew Luxton-Reilly, a significant contributor to the international CER community with several senior roles, including membership on the ACM SIGCSE Executive Board; Lillian N. Cassel, working currently on the ACM data science curriculum task force with numerous other interests and contributions within the computing and CER domains; Tiffany Barnes, her research focusing on AI for education, educational data mining, serious games for education, health, and energy, CER, and broadening participation; Gregory Hislop, with his current research focusing on the educational value of student participation in humanitarian free and open-source software (HFOSS) projects, a contributor to numerous initiatives and efforts in computing and CER; Heidi Ellis, a founding member of the HFOSS project, whose research interests include software engineering education, open-source software, and tools for biological data analysis; Peter Hubwieser, whose research focuses on empirical investigation of learning processes in computer science (definition, measurement, and evaluation of competencies, skills, and knowledge structures); Orit Hazzan, with her research focusing in computer science, software engineering and data science education; Renée A. McCauley, her scholarly activities revolving around computer science education, and including: teaching, curriculum development, and collaborative education-based research; Judith Gal-Ezer, her research in CER focusing, among other topics, on algorithmic thinking, the teaching of subjects such as recursion, complexity and efficiency, misconceptions in computer science, reduction and non-determinism, Dan Garcia, a leader of the "CSforALL" movement, winner of numerous awards, having served on numerous boards such as the ACM Education Board, the College Board Computer Science Principles Development Committee, and the most frequent SIGCSE author in its 50-year history with over 61 submissions; Kristy Boyer, with her research focusing on CER, AI in education, intelligent tutoring systems and dialogue systems; Kate Sanders, with her research interest in empirical CER, a participant in the bootstrapping CER workshop in the 2002-2003, and a prominent author in SIGCSE, ITICSE, and ICER.

C. COUNTRIES
The global distribution of CER, shown in Fig. 8 shows strong USA dominance in CER. The top ten contributing countries in CER are USA, Finland, Germany, Denmark, Australia, UK, Spain, Sweden, Canada and New Zealand.

VI. THE DISSEMINATION OF CER
Dissemination (Lat. disseminare, "scattering seeds") means broadcasting a message from a sender to a number of receivers. In the early days of CER, outlets such as SIGCSE Technical Symposium mostly published experience reports, while new venues in the 2000s, such as ICER started to demand papers with a clear theoretical base, which draw on previous research, and share a strong empirical basis. The venues of dissemination, and their habits and cultures play a crucial part in the evolution of a scientific discipline. From an author's perspective, the selection of a suitable publication venue is guided by a set of attributes including: intended audience, reputation of the venue, turnaround time, acceptance rates, page limits, citation metrics, and institutional rankings.

A. TOP 20 PUBLICATION OUTLETS OF CER
Our dataset included CER publications from 1840 distinct sources. Table 3 shows the top 20 publication outlets in the data with regards to numbers of articles published. Some 68% of publications in our dataset were published in the top 20 outlets, and those articles have attracted some 73% of all citations in the dataset. The top 20 publication outlets (Table   14 VOLUME 4, 2016 3) include the well-known venues that exclusively publish CER, introduced in Sec. II.
Top 20 venues (Table 3) include the two major journals of CER, Computer Science Education (CSE; row 7) and ACM Transactions on Computing Education (TOCE; row 12), formerly known as Journal on Educational Resources in Computing (JERIC, row 14), which was a pioneering venue in advancing tools-research in CER. ACM TOCE (row 12) has the second highest mean citation rate among the top 20 venues, and only 1% of its articles have been never cited. ACM JERIC (row 14) also has a high mean citation rate of 20.5. The mean citation rate of CSE is 11.9 (Table 3). With regards to conference proceedings, the ACM's Special Interest Group in Computer Science Education's (SIGCSE) Technical Symposium (row 1) has by far published the largest share of CER (some 23%) in our dataset. Other well-known CER outlets include ITiCSE (row 2), ACE (row 11), Koli Calling (row 6), and ICER (row 5). SIGCSE, ITiCSE, Koli Calling, ICER, and ACE form the core of conferences, which are dedicated to exclusively publishing CER [7], [14], [55]. The new CSERC conference is found on row 19, and the two well-known conferences dedicated to publishing K-12 research, ISSEP and WIPSCE are found on rows 8 and 13.
With regards to other venues, where CER is known to be occasionally published [7], our data shows publications in  [55].
In addition to these well-known venues, the top 20 venues (

B. PUBLICATION AND CITATION PROFILES
The articles in our dataset were spread over a variety of venues. Eleven of the top 20 venues are dedicated to ex-   CER is inherited from the CS discipline, where many conferences are also highly regarded, in contrast to many other disciplines of science, where journals are more highly regarded. Our analysis shows that journal articles of CER receive significantly more citations than conference articles, with an average of 11.92 citations (journal article) versus 6.72 citations (conference article), (t(3844.79) = 9.15, p < .001, Welch's t-test, two-tailed). So, while people tend to publish CER in conferences, journal articles are cited significantly more. Fig. 9 shows the citation profiles of the top 20 CER venues. Fig 9, together with Table 3 shows that in SIGCSE and ITiCSE, which are the most influential publication outlets, most articles have received only a small number of citations (M = 4, SIGCSE; M = 2, ITiCSE), with a few highly cited star papers, while e.g. ACM TOCE, and IEEE ToE have healthier distributions of citations (M = 10, TOCE; M = 9.5 in IEEE ToE). Some 23% of all articles in the top 20 venues have never been cited, while some 58% of all articles in the top 20 venues have been cited for five or fewer times (including those with zero citations). For some of the venues, such as ACE and ISSEP, some 69% of articles have received five or fewer citations. For SIGCSE, which is by far the most popular venue in CER, some 57% of articles have received fewer than 5 citations and some 23% have never been cited. The modest citation rates in CER research have been observed also in previous research [45].
For comparison, in Computers & Education, which was the top 26 venue of CER publications in our data, the mean and median for publications were 42.8 and 14, with some 25% of articles being cited five or fewer times and 11.1% of articles never being cited. Publisher-wise, dissemination of CER is dominated by ACM (see Fig. 9), with 14 of the top 20 venues published through ACM.

A. KEYWORD TRENDS (RQ1)
Our first research question asked: "How has the publication profile of CER evolved in terms of keyword trends and topics of research? (RQ 1 )" Our analysis of the most cited papers in the active six decades of CER, as well as the analysis of absolute, relative, and ranked keyword trends post 2000, reveals several things.
First, the most cited papers in the past decades connect well with the narrative of the evolution of CER [7]. The times before the 1970s were about laying the groundwork for CER. Publications focused on new curriculum guidelines and recommendations for the then-new discipline of computing. The 1970s were about seeking for identity with increased focus on hands-on work and applications, tools, pedagogical aids and course descriptions. The 1980s saw the emergence of empirical research, and the entry of new topics in software engineering. The 1990s demonstrated an increase in pedagogical considerations in CER, and the 2000s were marked by continuing dominance of teaching computer programming, new topics such as program visualization, and increase in empirical research. The 2010s are characterised by CER maturing as an independent field of study, and the increase of research in K-12 and computational thinking. Cluster analysis revealed two keyword-clusters formed around programming-related topics.
Second, analysis of the most cited papers, and analysis of keyword trends show that teaching and learning programming has by far been the all-time most popular topic of CER. The roots of programming as a central topic in CER are deep. Starting from the first programming textbooks in 1951, through the emergence and growth of the software industry in the 1960s, to the influential ACM curriculum recommendations in 1978, programming has had a strong foothold in CER [64]. A central debate in both the 1970s and 1980s was how to teach programming [7], [69]. In the 1980s also empirical research on programming started to appear [73], and in the 1990s, programming continued to be a central topic [76], [79]. The 2000s saw an ever increasing quantity of research and seminal papers on teaching and learning programming [8], [81], [82]. The trend continued into the 2010s, and the strong emphasis on programming is also clearly visible in previous analyses of publications in many central venues of CER [1], [15], [21], [22], [45].
The dominance of programming warrants some reflections. Programming is, in many ways, at the core of computing, making it an important topic of research. On the other hand, the typical CS curriculum is filled with many topics of equal importance to programming and e.g. data structures, such as theory of computation, databases, machine learning, and concurrency. This may make one wonder: why is CER so concentrated in first-year courses on introductory programming? A number of voices are stressing the importance of CER on topics such as machine learning [119]- [121], design research [109], [122], [123], and STEAM integration [124]- [126]. Indeed, while entire generations of people are growing up in the middle of machine learning (ML) systems, this development seems to have been given only minor attention in CER. A small but growing body of research shows concrete examples of teaching ML to beginners [119], [120], [127], [128]. New social and ethical dilemmas created by new AIsystems also call for reshaping of related training in AI ethics [129], [130].
Other voices call for CER on rigorous pedagogies to add understanding of communities, habits, and cultures into the technology development process [109], [122], [123], [130]. Indeed, it has been argued that future crucial breakthroughs will increasingly not only be programming breakthroughs, but increasingly design breakthroughs [109], [122], [123], and that technology innovation requires practitioners wellversed in techno-solutionist methodologies, but increasingly also in social realities to a much larger extent than can be done with shallow methods of, e.g., user requirement definition [130]. While ML, design, and ethics have been niche areas in CER, their relevance is increasing rapidly. In addition, while the need for CER on ML is increasing, this also increases the relevance of CER on topics such as 16 VOLUME 4, 2016 basic probability and statistics, currently almost non-existent [119]. While nearly the entirety of publications on CER focus on classical programming, research on how people learn ML and design is needed, not to mention the other underresearched topics [109], [119], [130]. We hope to see increasing trends of CER on these topics in the near future.
Third, keyword trends in the past two decades indicate that, in addition to programming, computational thinking (CT) and K-12 are top trends. Indeed, the keyword analysis shows that the keywords "K-12" and "computational thinking" (CT) are among the most popular keywords in the CER publications, with a significant increase in popularity over the past decade. While the skyrocketing trends of CT and K-12 were launched in 2006 after the publication of the seminal paper of Jeanette Wing in 2006 [91], the roots of CT are in the 1950s in the work of, e.g., Donald Knuth, Edsger Dijkstra, and many others [106], [131], and, e.g., in Seymoyr Papert's groundbreaking Mindstorms [103]. Other common labels for CT are, e.g., algorithmic thinking, computational making, and computational participation [132], [133]. After CT was made popular by Wing in 2006 [91], the number of CT-related publications began to grow, with the annual output of CT publications accelerating rapidly, reaching some 430 articles in 2019, with an annual percentage growth rate of 61.2% [48]. A wide repertoire of approaches for computing in K-12 exist, many of them focused on programming, with some 27.2% of CT articles including programming-and coding-related terms [48]. A common approach is blockbased programming with Scratch [134], and educational robotics [135]. Also in the context of CT, a large amount of CER seems to concentrate on rule-driven programming or, e.g., logic puzzles. Future recommendations [136] include changing the public misconception of "computer science = programming", changing the common stereotype that only social misfits can do programming, increasing basic training on machine learning in CT [120], and abandoning logic puzzles in favour of well-established and brilliant pedagogical toolkits, such as the CS-Unplugged [137], [138].
Fourth, in addition to programming and computational thinking, the top topics of research include gender and diversity, assessment, and learning analytics & educational data mining. Other keywords included in the top 20 keywords, include pedagogical considerations such as collaboration in learning, and research on software engineering education, which has been a fundamental part of CER throughout the years. Also, research on educational technologies, gamebased learning, and visualization are strongly visible in the top keywords. Research on educational technologies, games, and visualization tools all belong to the category of toolsresearch within CER.
From early on, an important category of CER has been that of tools and pedagogical aids, and this is reflected in the cluster analysis of keywords, especially in the red cluster with the focus on educational technologies, automatic assessment and e-learning. A survey of CER articles published prior to 1977 shows that reports on tools, pedagogical aids and course descriptions were common, while empirical research was rare [67]. Empirical research started to appear in the 1980s, e.g., in the Psychology of Programming Interest Group (PPIG) publications, and through the influential work of Elliot Soloway [69]. While empirical research is nowadays a fundamental part of CER, research within the tools category has remained equally important. System papers and experience reports feed research by presenting innovations and allowing the community to familiarize itself with novel ideas even before any more rigorous evaluations have been carried out. A crucial trajectory of CER is that of tools: from early diagnostic compilers [66] to the automatic assessment and visualization tools of today, as is demonstrated by the top keywords and keyword clusters.

B. BUILDING BLOCKS OF CER (RQ2)
Our second research question asked: "What do citation and co-citation metrics reveal about the foundational work in the CER discipline? (RQ 2 )" Our cluster analysis of cocited papers shows several things with regard to how CER researchers build on previous work.
First, clusters of foundational work-articles that researchers of CER are likely to cite in their research-were formed around the most researched areas of CER. Two clusters were formed around teaching and learning programming, with their own separate orientations. While the first cluster on programming (green, traditional track) includes a combination of influential reviews and works, it also includes a set of classical works on education and learning, namely the cognitive load theory, Bloom's taxonomy, SOLO taxonomy, and constructivism. Bloom's taxonomy, cognitive load theory, SOLO taxonomy, and self-efficacy theories have been identified as much used learning theories in CER [33], [36]. The second cluster on programming (orange, the modern track), contains no theoretical contributions on learning outside of CER. Instead, it includes pedagogical approaches designed by and for the purposes of learning and teaching programming, by CER researchers, such as methods called "extreme apprenticeship" and pair-programming.
Second, while the first cluster on programming leaned more heavily on classical work in education, the second cluster was more oriented towards developing its own CERspecific constructs and pedagogies. Previous meta-work has investigated how CER builds on previous theories and models and how CER develops its own theoretical constructs (TC) [34]- [36], [139]. Findings from a recent review, looking at CER from the viewpoint of influential learning theories, identified Csikszentmihalyi's flow theory, learning styles theory, self-efficacy, problem-based learning, and communities of practice as the most used, despite having an overall limited reach within CER [36]. Other findings show that CERspecific TC's are being developed in the areas of emotions, attitudes, self-efficacy, and other areas [139], and cohorts of articles are emerging that use these constructs as their base [34], [35]. Also, much previous work seem to draw on education and psychology [34]. Overall, a modest use of TCs VOLUME 4, 2016 has been found, and in many cases TCs were merely mentioned in citing articles with no clear connection or deeper application of the TCs [35]. The findings, on one hand, show modest use of theories and background with no prevailing theoretical works that are broadly applied and, on the other hand, the maturation of CER, building its own theoretical constructs and in this way claiming its independence [34], [35], [139].
A somewhat restricted set of learning-theory use has been observed in previous reviews [36], and complemented by our findings in this paper. Previous research has also identified widespread use of outdated and largely debunked learning theories, such as the learning styles theory, within CER. While development of CER-specific theories of learning is important, it is also important that researchers build their work upon previous research. It seems fair to recommend that CER researchers become better informed of the rich repertoire of available learning theories (eg. [36]), aim to avoid debunked and outdated theories, and do not build their research only upon the most commonly used learning theories. New and innovative openings to CER may be found by learning to appreciate the richness of available theoretical contributions.
Third, one major (pink) cluster was formed around computational thinking and K-12. This cluster includes a heavy presence of foundational tools and pedagogies in CER, including LEGO Mindstorms, Alice, Scratch, and CS-Unplugged. In addition to the most co-cited works in the K-12 and CT domains, revealed by our analysis, there exists other influential contributions in this domain, too. Indeed, it is important to ask, why does a majority of researchers cocite these specific works and not some others. Other influential contributions include those authored by e.g. Denning, Dagiene, Tedre, Mannila and Pears [106]- [109], and from the times before Wing's discussion paper in 2006 [91], influential works include those of e.g. Knuth, Dijkstra, and many others [106], [131]. While Wing's paper was not a research paper but a discussion paper, it marked the starting point for the new trend of CT and K-12 research, and has gained enormous popularity, and high numbers of cites and co-cites. But it is indeed relevant to ask, to which extent the majority of CER researchers on this domain are aware of other influential works beyond the most co-cited ones, and what could be done to increase the community's awareness? A recent analysis about the publication trends in CT [48] delves in this theme with more depth. One possible explanation for amplified citations to certain works is the Matthew effect [140], which can be summarized as "the rich get richer and poor get poorer", and may explain the heightened visibility of contributions by researchers of acknowledged standing and reduced visibility of contributions by other authors [140]. Moreover, the amplifying of already cited research by some search engines may generate echo chambers, which lead to researchers overlooking other influential works, while concentrating on citing certain limited sets of recommended and much cited previous works.
Fourth, the smaller gray cluster represents foundations in research that deals with diversity, gender, and social aspects. This cluster includes a set of educational work including that of Vygotsky, Bandura's self-efficacy, and constructivism. Self-efficacy and constructivism have been identified as common learning theories in CER [33], [36]. Smaller clusters represent work on social aspects and diversity, automatic assessment, meta-research and emotions. Self-efficacy and constructivism were also found to be the most commonly cited learning theories in CER [36].
Fifth, only a single contribution on research methodologies was found in the foundational papers, in the computational thinking and K-12 cluster. Neither of the clusters on computer programming had any methodological contributions. Previous meta-research has looked into methodology use in CER papers [28], [33], [38], and found, e.g., that many papers make use of statistical techniques beyond descriptive statistics [28], and that commonly used statistical tests include the t-test, χ 2 -test and Mann-Whitney-Wilcoxon -test [38]. Our analysis found only one methodological contribution, Cohen's statistical power analysis [105], which introduces, e.g., Cohen's d for calculating effect sizes. This is in contrast to educational technology research, where methodological contributions are much more widely referred [93]. One explanation for the low number of methodological contributions is that CER researchers may be uneducated on the literature on research designs and research methods. Another explanation is that while statistical methods, and research designs may be commonly used in CER, citations for them are not widely presented. A recent review of CER published in 2014-2015 [37] supports the latter assumption, when it comes to CER published in the past decade. While quantitative evaluation methods are used by CER researchers, and papers frequently report results on pedagogies, curriculums, and tools, many papers lack properly reported research objectives, goals, research questions, or hypotheses, description of participants, study design, data collection and threats to validity [37]. It seems that while CER researchers are, at least in recent times, actively using research methods to conduct empirical studies, not all norms of reporting are fully met, including those of research design and research methods, as well as proper citations of methodology work.
The ad-hoc nature and lack of research rigor in CER begun to attract a lot of attention in the beginning of 2000s [141], [142]. It became well known that a large share of CER concentrated on describing course contexts and teaching practices in an anecdotal manner, and the need to increase methodological rigor, build connections to learning theories, and need to expand qualitative research became evident [141]- [144]. A series of workshops to educate academics with a computing background on how to design and conduct CER, conduct qualitative research such as phenomenography, and proposals for CER-specific research frameworks started to emerge [13], [141]- [144]. In 2008, an increasing number of papers with increased methodological rigor was observed in CER conferences [144]. While the value of 18 VOLUME 4, 2016 empirical research has been on the increase for a while, it is relevant to highlight the important focus of CER in addressing and solving pragmatic challenges related to teaching and learning [145]. The series of research on developing research frameworks of CER, and on how to arrange workshops to train methodology to CER researchers forms an important part of CER. Future scientometric studies and reviews could have a special focus on this important series of articles, which not only categorises previous research with regards to methodology use, but, which focuses on improving the situation through implementing various workshops and training on CER methodology.

C. CREATORS OF CER (RQ3)
Our third research question asked: "How have the knowledge creators (authors and institutions) and their collaboration shaped the discipline of CER and its communities? (RQ 3 )" Our analysis reveals the following.
First, our findings from analyzing the institution-level collaboration networks show the presence of an elite group of 52 institutions, which collaborate extensively with each other, and who conduct and publish a remarkable share of CER. The share of articles published by the rich core attracts significantly more citations than other articles in the dataset. Some in the research community might find such inequality alarming. On the other hand, it is likely that the success of the rich-club institutions has resulted from their ability to develop broad expertise and partnerships, and excel in both variety and quality of research. An important question is: to what extent do the core of elite universities have a wider impact on driving science forward, and to what extent are the less successful institutions able to benefit from their associations with the institutions of the rich core?
Research as an enterprise is fundamentally driven by collaborative relations and their dynamics. Previous research has shown cases in which elite circles of academic institutions overattract research funding, while at the same time they collaborate with members of the very same elite circles [117]. Membership in a "rich club" of academic institutions may offer easier access to other elite members, give strategic advantages as compared to non-elite members, and provide the elites with the potency to boost their power by controlling access to opportunities [117], [146]. In short, research funding may mainly go to rich clubs of science, who have control over topics and scholarships and who decide which research is valued, and which is not, potentially causing inequalities and discrimination. In this sense, future research should zoom in on the research produced by the rich core, identify leading trends and themes of research, the degree of interdisciplinary research, and the quality of collaborations with institutions that remain outside of the rich core.
Second, our findings show a diverse set of influential authors with an equal balance of female and male researchers, working on a diverse set of topics, and pushing the boundaries of CER. Third, geographical distribution of CER shows strong USA dominance, followed by a selection of high-income countries. The dominance of high-income countries in CER, especially USA, has been observed in a number of previous research studies [42]- [45], too. Also, previous research has shown that while collaboration in CER is growing, it still mostly happens within the same country [42]. It seems that only a few papers originate from, or address challenges of computing education in the Global South, a situation observed already back in 2010 [147]. For example, research from Africa, a home to 1.2 billion people, is rarely seen. The CER community needs to act in order to better serve the needs of computing education for all, in all countries, not just in a predominantly Western few.

D. DISSEMINATION CER (RQ4)
Our fourth research question asked: "How can the central venues of dissemination of CER be characterised with regards to their publication profiles and citation practices? (RQ 4 )" Our analysis shows the following. First, the top 20 publication outlets include all the well-known publication outlets of CER, and together they publish some 68% of all CER, attracting some 73% of all citations. Other common venues that publish CER are from closely connected disciplines, such as IT education, general computer science, engineering education, or general education. Second, a major share of CER is published in conferences. By far, the most influential venue with regards to numbers of publications is the SIGCSE Technical Symposium. The tendency to publish in conferences is inherited from the CS discipline. A major share of the CER conferences are published by ACM. Third, while CER is often published in conferences, journal articles in CER are significantly better cited. Fourth, many venues have a small group of highly cited articles and a large group of minimally cited articles. A healthier citation profile is found, e.g., in the ACM TOCE. Fifth, CER suffers from modest overall citation rates. A large share of CER is never cited, and over half of all CER is cited five or fewer times. Similar observations have been found in previous research, too [45]. One exception is CER published in Computers & Education, attracting a significantly higher number of citations than in any of the top 20 CER venues. Of course, citations are not the only measure of impact, as many articles may be read by interested educational practitioners. In any case, it is good to ask if the current situation is desirable, and if not, what the CER community can do to address the issue.

E. FUTURE RESEARCH
A number of avenues for future research exist. While this research covered CER on a wide range of topics, areas, authors, and geographical areas, more focused scientometric reviews could be conducted to gain deeper insights into specific important research niches, work of specific communities of authors, and work in geographical areas that are currently not given attention. For example, future research could focus on methodology development literature in CER, or they could focus on underrepresented researchers or geographical locations, such as those of the Global South, to VOLUME 4, 2016 reveal patterns of research, important and missing topics, and patterns of collaboration. Indeed, while the field of CER has been heavily dominated by USA-originated research, and also topics such as teaching classical programming, it would be crucially important to focus future studies on topics of improved importance, and on underrepresented geographical areas. Future research should also investigate research produced by the rich core, identify leading trends and themes, degrees of interdisciplinary research, and look into the degree and quality of collaborations with institutions outside the rich core. Dissemination with the publication and citation practices forms another important avenue for future research.

F. LIMITATIONS
The following limitations apply to this research. First, several caveats apply to the data. While Elsevier's Scopus database is well maintained and, in most cases, more accurate than WoS (Web of Science) [148], it is not perfect. The earlier years especially have some issues: missing fields, inconsistent and unstructured keywords, references not perfectly recorded, and mistakes in publication venue names. As one example, an influential article from 2003 [10], published in the journal Computer Science Education, is misclassified in Scopus as being published in International Journal of Phytoremediation. Another example is reviews that fail to be classified as reviews, and therefore get included in the dataset. Even though we have manually checked author and venue names, used manual and algorithmic methods for combining keywords, and given our best efforts to fix inconsistencies in the databases, detecting and manually correcting all such mistakes is not possible. Data prior to 2000 were found more vulnerable to mistakes and mis-classifications. For example, keywords use was very inconsistent before 2000, and therefore we have conducted keyword analysis only post 2000. Data prior to 1970 were unreliable and needed to be subjected to many manual checkups. In all, by including data from the well-known dedicated venues, carefully controlling the articles retrieved through keywords searches, and with extensive cleaning of the data, we have reached a representative, if not comprehensive, sample of CER.
Second, scientometric methods provide a quantitative view and remain shallow without a qualitative perspective. Scientometrics as a method is severely limited as compared to systematic reviews and meta-reviews, which can capture a number of important perspectives that are out of reach for scientometrics. With scientometrics, it is not possible to analyze methodological rigor, how learning theories are exactly applied, the research designs, the empirical or experimental nature of the work (if not available in article metadata), or the replicability of studies. On the other hand, reviews are typically restricted to a few venues with a limited timeline, and cannot provide a holistic analysis of trends, themes, communities, venues, and their evolution, which can be done with scientometrics. Indeed, both of these two approaches, scientometrics and systematic reviews, are needed, and they complement each other in building a holistic picture of the evolving discipline of CER. In order to turn the numbers into meaning, we have included a combination of perspectives, and involved experts with decades of experience in the field to create an understanding of things.

VIII. CONCLUSIONS
Over the recent decades, computing education research (CER) has matured into a respectable field of research. New conferences and venues have been formed, methodological quality and rigor has improved, own discipline-specific constructs and pedagogies have been developed, and experience reports and tool-papers have been complemented with an increasing amount of empirical research. While CER has become more international, it is still dominated by few highincome countries, especially the USA. As teaching and learning computer programming still strongly dominates CER as a research topic, it is necessary to reflect on the need for diversifying the common themes of research. Separate threads of CER build on traditional learning theories as well as developing their own discipline-specific pedagogies and constructs of learning. The lack of citations of research methods raises the issue of how to increase the methodological knowledge and skills of CER researchers. While the discipline of CER is led by an elite group of 52 institutions, it is relevant to reflect to what extent the less successful institutions are able to benefit from their association with the rich core. Finally, the low citation rates in many central venues of dissemination call for a response from the CER community.
MIKKO APIOLA holds a PhD in computer science from the University of Helsinki, Finland. He has work experience in IT industry as sofware developer and IT consultant, and has worked in several teaching and research positions in Finnish universities. His research interests include computing education research, learning analytics, ICT for development (ICT4D), and educational technology. He currently holds positions of Adjunct Professor at the University of Turku's Department of Computing, Finland, as well as at the University of Eastern Finland's School of Computing.
MOHAMMED SAQR has a PhD in learning analytics from Stockholm University. He works as a senior researcher at the University of Eastern Finland on Artificial intelligence, big data in education, network science and scientometrics. He is particularly interested in research methods including network analysis, temporal networks, machine learning, process, and sequence mining as well as temporal processes in general. He is also an active member of several scientific organizations and acts as an academic editor in leading academic publications.
SONSOLES LÓPEZ-PERNAS (GSM '19) completed her Ph.D. in telematics engineering at Universidad Politécnica de Madrid (UPM), where she also received her Bachelor and Master's degrees in telecommunications engineering. Since 2015, she has worked as a researcher with the Next Generation Internet Group (GING) at UPM. She is currently an Assistant Professor in the Department of Computer Science in the same institution. Her research interests include learning analytics, technology enhanced learning, computer science education, and bibliometrics.