From a National Meeting to an International Conference: A Scientometric Case Study of a Finnish Computing Education Conference

Computerisation and digitalisation are shaping the world in fundamental and unpredictable ways, which highlights the importance of computing education research (CER). As part of understanding the roots of CER, it is crucial to investigate the evolution of CER as a research discipline. In this paper we present a case study of a Finnish CER conference called Koli Calling, which was launched in 2001, and which has become a central publication venue of CER. We use data from 2001 to 2020, and investigate the evolution of Koli Calling’s scholarly communities and zoom in on it’s publication habits and internalisation process. We explore the narrative of the development and scholarly agenda behind changes in the conference submission categories from the perspective of some of the conference chairs over the years. We then take a qualitative perspective, analysing the conference publications based on a comprehensive bibliometric analysis. The outcomes include classiﬁcation of important research clusters of authors in the community of conference contributors. Interestingly, we ﬁnd traces of important events in the historical development of CER. In particular, we ﬁnd clusters emerging from speciﬁc research capacity building initiatives and we can trace how these connect research spanning the world CER community from Finland to Sweden and then further to the USA, Australia and New Zealand. This paper makes a strategic contribution to the evolution of CER as a research discipline, from the perspective of one central event and publication venue, providing a broad perspective on the role of the conference in connecting research clusters and establishing an international research community. This work contributes insights to researchers in one speciﬁc CER community and how they shape the future of computing education


I. INTRODUCTION
Koli 1 Calling is an international conference for computing education research, which has been arranged annually since 2001 in a resort in the Koli National Park, close to Joensuu, Finland.Over the years, Koli Calling has expanded The associate editor coordinating the review of this manuscript and approving it for publication was Rebecca Strachan . 1 Koli is a mountain by the lake Pielinen in municipality of Lieksa in Eastern Finland.It mainly consists of white quartzite, and its summit has large treeless areas.Koli village is at the root of the hill, and the area is part of Koli National Park.(Wikipedia) from a local conference for computing educators mainly from Finland and its neighboring countries to a respected international venue for computing education research.As the conference has evolved over the years and as its community has diversified, the history of the now 20-year old conference offers an opportunity to study the evolution of the conference's community, topics, and the community's publication and citation habits.
Our research questions are: • How have authors and author networks shaped Koli Calling and its community over time?
• How has the publication profile of Koli Calling evolved in terms of most-cited papers, keyword trends and keyword clusters?
• How has Koli Calling evolved from the viewpoint of international collaboration?Previous studies, overviews, and meta-research of Koli Calling have shown a vibrant and evolving community with a special atmosphere and sense of community, evolving patterns of contribution, maturing publishing profile, diversifying authorship, and international collaboration [1]- [4].The purpose of this study is to expand the previous accounts with a bibliometric perspective in order to explore the Koli Calling conference with regard to collaboration networks, shifts in research focus, international and national collaboration, and citation metrics.The goal is to provide a historical analysis that allows the conference to strategically position itself within the field as it continues to develop an international profile.Koli Calling is one of the dedicated publication outlets of CER, together with two active journals and a handful of conferences [5].CER is evolving from publishing experience reports and anecdotal evidence to empirical research and rigorous use of methods, and the venues of dissemination play a crucial role in the evolution of CER [5].CER is becoming a respectable research specialization with its established professorships and publication outlets [5].It is crucial to subject the main publication outlets of CER to research in order to understand how CER evolves, and how can the publication outlets better serve career advancement in CER.
The availability of well-structured databases and indices of research such as Scopus and Web of Science, together with modern data science and network analysis methods have resulted in increasing popularity of bibliometric studies in recent years [6], [7].The scope of these new tools and methods go far beyond simple counts or descriptive statistics.The modern scientometric and bibliometric tools provide researchers with means for transparent and reproducible studies of science and publications.Bibliometrics provide a nuanced overview of the evolution of knowledge production over time [6], [7].In this research, state-of-the-art bibliometric methods are applied to offer a quantitative, indepth view of the history of the Koli Calling conference and community.This article is structured as follows.First, background is presented in Sec II, followed by related research in Sec III and methods in Sec IV.Then, the results of the analyses are presented in the following manner.Analysis of authors is presented in Sec V, and analysis of collaboration networks in Sec VI.This is followed by analysis of influential articles (Sec VII), international collaboration (Sec VIII), and keywords (Sec IX).Finally, the results are discussed (Sec X).The article is concluded in Sec XI.

II. BACKGROUND
The first Koli Calling conference was organized in 2001 at the initiative of Professor Erkki Sutinen from University of Joensuu, later the Joensuu campus of the University of Eastern Finland.Initially, the event targeted Finnish computer science educators in universities, and it was designed to be a fairly informal meeting for educators to share ideas and experiences and in some way shake-up the traditional ways of teaching and studying computer science [8].The first Koli Calling conference had 15 paper presentations, which gave plenty of room for many open sessions aimed at community building and constructive critical discussions.
For the next two years, Sutinen continued chairing the conference, and as the number of foreign participants gradually increased, the session language changed between Finnish and English, depending on whether everyone present in the session understood Finnish or not.The attendance turned gradually more international-the first years saw Moti (Mordechai) Ben-Ari (Weizmann Institute of Science, Israel), Anders Berglund (Uppsala University, Sweden), and Andrea Valente (Aalborg University, Denmark)-but for the first years, the focus remained national and in many ways not unlike the many other ''swap meets'' of computing educators to share their experiences [9], [10].For the first years the number of papers remained stable (Fig. 1).
In 2004, the then-small program committee decided to start developing the conference profile towards an international research conference.For that purpose, the call for papers was revised and disseminated internationally, and the conference now solicited submissions in three categories: research papers (8 pages), discussion papers (4 pages) and posters/demos.The decision to separate the first two categories was intentional.Research papers should have sufficient space within which to properly report research, whereas discussion papers should report ''work in progress'', or dissemination and discussion of new ideas within the CS education community.
The conference recognized the diversity of work done in the CER field: For instance, Valentine's study of Special Interest Group in Computer Science Education's Technical Symposium (SIGCSE TS) proceedings found the SIGCSE conference to be a grab bag of experimental research, experience reports, analytical papers, tools articles, nifty assignment descriptions, and ''John Henry'' papers [10].After 2004, Koli Calling conference took on the task to promote the diversity of work carried out in the field and avoid value judgments between different types of research and development.Research papers continued to be the format for reporting empirical research, but new categories were launched to solicit other types of contributions.
The new system papers category, introduced in 2005, supported publishing tools research.The program committee noted that many educators and researchers developed new, interesting educational software tools tailored to support computing education, such as tools for automatic assessment, visualization, or simulation (cf.[11]).Developing such tools takes considerable time and intellectual input, but it was getting increasingly difficult to publish such work without an empirical evaluation.The system papers category provided a way to present an educational tool, its use cases and pedagogical rationale, and thus present the tool for the community long before a comprehensive empirical evaluation would be ready.In some years, Koli Calling even had a separate call for tools which called for a report, but also required the authors to provide the actual software for pedagogical evaluation.
Another new category, theoretical research papers, recognized work that develops a solid argument based on literature and/or existing theoretical frameworks in order to address a problem relevant to computing education without a rigorous empirical evaluation.The new categories implied that the reviewing criteria had to be revised to communicate the nature of the categories better to both authors and reviewers.While the category names have changed several times over the 20 years, Koli Calling has always wanted to support the richness of educational innovations in addition to presenting high quality research.
Community building has always been an explicit goal of Koli Calling [1], facilitated by several conference practices.Firstly, the conference is located at a hotel in the Koli national park, a unique place in Eastern Finland far from anything else.That means that all participants are available for social meetings after the paper sessions, instead of disappearing into restaurants and city attractions in the evenings.Secondly, the venue limits the conference size to 50-60 people (the number of papers remained around 20-30 from 2004 to 2016; see Fig. 1).The compact size gives an opportunity to see and meet practically everyone during the event from Thursday evening until Sunday morning.This is highly valuable for PhD students who can easily and informally reach seniors to get acquainted with them and discuss their own work and ideas.
Thirdly, the conference is organized in a small classroom where people are very close to each other and the speaker, instead of being spread out in a large lecture hall.This sense of closeness promotes positive and constructive feedback for presentations [1].Finally, the social program has its specialties: Saturday afternoon's walk in the national park is a highlight for many visitors, especially if there is snow, which happens in most years.Koli sauna party and Koli Spa visit also give a very relaxed setting to meet others and learn to know new people.

III. RELATED RESEARCH
The most well-known conference that exclusively focuses on CER is the ACM's Special Interest Group in Computer Science Education's (SIGCSE) annual symposium, which was launched in 1970 [12], [13].In 1996, SIGCSE was joined by its European counterpart ITiCSE (Innovation and Technology in Computer Science Education), while ACE (Australasian Computing Education) was launched in 1996, Koli Calling in 2001, and ICER (International Computing Education Research conference) in 2005 [5], [12].ACM's new Global Computing Education Conference (CompED) was established to serve scholars outside North America and Europe, and was arranged for the first time in 2019 [5].SIGCSE, ITiCSE, Koli Calling, ICER, CompED, and ACE are well-known conferences that focus exclusively in publishing CER [5], [12]- [14].
A number of research and meta-research have analysed or meta-analysed publications in the known CER publication venues (e.g.[1], [5], [15]- [18]).One of the earliest efforts to analyse CER communities was the analysis by Valentine, which focused on SIGCSE Technical Symposium [10], and the classifications Marco Polo, Tools, Experimental, Nifty, Philosophy and John Henry.Valentine's analysis led to a number of other efforts to analyse CER publications, both nearly immediately [19], and in the years that followed [9], [14], [15], [20].Recent research to classify CER publications has classified publications from SIGCSE and ITiCSE conferences [21].Scientometric analyses of CER publications are also starting to emerge, including analysis of collaboration networks in SIGCSE, ITiCSE and ICER [22], analysis of the geographical diversity of the SIGCSE TS [23], and an investigation into the publications in ASEE/IEEE Frontiers in Education conference, which is dedicated to both computing education research and engineering education research [24].In one research [25] the keywords in publications of ICER and ITiCSE were mapped, showing a major stream of research on classical programming.Research in computational thinking was mapped in a metrical study [26], while another research mapped the entirety of CER from the viewpoints of foundational works, keyword trends, institutional networks, and dissemination practices [5].However, no previous scientometric studies have analysed the Koli Calling conference by using modern scientometric methods.

IV. METHODOLOGY
The article metadata were retrieved from the Scopus database in Autumn 2021.The database contained the metadata of all articles in Koli Calling proceedings in the years 2006, 2008 and 2010 through 2020.The metadata of the articles published in 2007 were extracted from the ACM Digital Library, excluding posters.Metadata for the remaining years were manually extracted from the archived proceedings provided by the conference website (2009) and the organizers themselves (2001)(2002)(2003)(2004)(2005).Years 2001-2005, 2007, and 2009 lacked citation data as the conference proceedings were not indexed.The metadata from all proceedings articles were combined and cleaned, so author names, affiliations, and database inconsistencies were checked, verified, and fixed.Data for authors whom we know had changed their names were joined.Similarly, keywords were cleaned and similar keywords grouped together.For instance, keywords such as (computer, computers); (collaboration, collaborative); (capstone, capstone project) were combined.The keywords (computer science education) and (computing education) were removed, since they cover all the papers.The data were analyzed using the Bibliometrix R! package [7].The analysis included descriptive statistics in which the count of documents, authors, article types, keywords, countries, citations (limited by Scopus index) and other statistics were calculated, tabulated, or plotted.
To study the evolution of research themes and the relationships between them, we constructed a co-occurrence network where keywords that co-occur in the same manuscript are considered linked.For the readability of relationships and labels, the network size was limited to the keywords that occurred in more than 10 documents.The keyword network was partitioned using Louvain modularity decomposition: keywords that co-occur frequently constitute a common theme and were assigned to the same community (colored similarly in the network plot).A similar network was created to study the collaboration between authors who authored more than five articles, by using fractional counting [27].Authors were considered linked if they co-authored a manuscript together, and like the keywords network, also the author network was partitioned using Louvain modularity decomposition to cluster authors who frequently collaborated together.Author country was extracted from the affiliation field in the database: the country of the first author's affiliation was considered the manuscript's country.

V. AUTHORS
The shift from a local conference to an international publication and meeting venue for computing educators is clearly visible in Koli Calling author profile (Fig. 4.) According to our data, since the first Koli Calling in 2001, until 2020, 619 unique author names have appeared in the Koli proceedings.Among the authors of Koli Calling papers, 430 (69.5%) appeared just once, and 85 (13.7%) twice, with 1.93 mean papers per author.For those authors who have appeared more than once in Koli Calling, the mean number of publications is 4.05.Several authors stand out for a large number of contributions to the Koli Calling conference series.All authors on the list of twenty most productive authors (Table 1) have authored or co-authored eight or more papers in Koli Calling.The largest number of paper authorships in one year is five: A. Berglund in 2007.The top positions on the list of most productive authors featured well-known computing education researchers with 14 or more contributions in 9 or more proceedings of Koli.Lauri Malmi was involved in 24 papers over Fig. 2 presents the time spans of publishing activity of the twenty most active authors in Koli Calling proceedings.

VI. COLLABORATION
Koli Calling has brought people together, and has maintained its remarkable community feeling that has always pervaded it [1].The co-authorships in the papers in Koli Calling form a sparse network with a number of more strongly connected clusters.Nodes in Fig. 3 present the authors with most coauthors (more than five unique collaborators) and edges between the nodes present co-authorships between them by using fractional counting [27].Unconnected nodes in Fig. 3 are active collaborators whose co-authors are not among the most active collaborators in Fig. 3. Some well identifiable clusters have formed around the authors of Koli Calling.
The most prominent green-colored cluster around seven o'clock in the figure is formed around well-known Finnish computing education researchers mainly from the capital area of Finland, including Lauri Malmi, Ari Korhonen, Päivi Kinnunen, Petri Ihantola (formerly Tenhunen) and Arto Hellas (formerly Vihavainen).The yellow cluster at 11 o'clock is centered around researchers from Eastern Finland, including the founder of Koli Calling, Erkki Sutinen, and researcher Ilkka Jormanainen, with a colleague Meurig Beynon from Warwick, UK.The light-green cluster is a Australian-Finnish cluster including Nick Falckner from Australia, Keith from Ireland, and Finnish computing educators Antti Knutas, Jari Porras, and Jouni Ikonen.The pink cluster is formed around well-known Finnish CER researcher Ville Isomöttönen from Central-Finland area, and known Swedish CER researcher Anna Eckerdal.The red cluster is another Finnish-Australian cluster centered around researchers in Turku Western-Finland area, and Monash University in Australia, while the light blue cluster is centered around Carsten Schulte from Paderborn, Germany.Most of the clusters include a strong Finnish presence.

VII. ARTICLES
From its inception the Koli Calling conference aimed at a diversity of paper types, and explicitly encouraged the publication of tool reports, analytical essays, discussion papers, and posters/demos alongside with empirical research papers.Diversity in research topics is visible in the list of the conference's most cited papers (see Table 2), where very different types of papers have attracted the attention of other researchers.
The most cited paper in Koli Calling [28] is a review study of automatic assessment in programming education.With 289 citations in Scopus, the paper has become a very popular reference in automatic assessment.The second most cited paper [29] is a historical overview of the development of computational thinking (CT).The paper has been a popular source for CT discussions (63 citations in Scopus).The third most cited paper [30] is a study about misconceptions in visual programming exercises.With 49 citations in Scopus, the paper has become an oft-cited reference in misconceptions research, a research track with increasing importance and popularity.The other popular papers include exploratory studies, tools papers, experiment reports, and discussion papers.
Many topics in Table 2 center around teaching of programming: a paper [30] investigated misconceptions in programming, while another paper [31] investigated reasons why students drop out of CS1.Code reading in teaching programming was researched in [32], and comparison of languages in teaching introductory programming in [33].Another common topic was tools: for visualizing parsons problems [34], program simulation [35], and eBook for data structures and algorithms [36].Gamification was the topic in [37].Educational perspectives were taken towards exploring the relevance of Bloom's Taxonomy in CSE [38], investigating the difficulty of the rainfall problem [39], and understanding of primitive and object variables [40].One popular paper investigated emotion detection from students' texts [41]; that paper can be said to belong to the domain of more generic educational technology research.

VIII. INTERNATIONAL COLLABORATION
Koli Calling started in 2001 as a local, national conference.Participants from the neighboring countries started to join in the following years.Fig. 4 shows the frequencies of contributions per country each year, as determined by the affiliations of the first authors.The number of articles from other countries has increased over the years, while the share of papers originating from Finland has steadily declined.In the first year of Koli Calling in 2001, all authors were associated with Finnish universities, but by 2006 the number of Finnish authors had shrunk to fewer than half, and in 2020, the largest number of authors came from the United States.Over its 20-year history, Koli Calling evolved from a local gathering into an international and intercontinental event and publication venue.A closer look at the 20 most active countries in Koli Calling proceedings shows that while papers have originated from all around the globe, the top contributing countries, based on the first author's affiliation in each article, are: Finland (373 papers), United States (101 papers), Germany (79 papers), United Kingdom (60 papers), Australia (50 papers), and Sweden (49 papers).Fig. 5 shows the distribution of articles in Koli Calling across the globe.Fig. 1 shows that although most papers in Koli Calling have been written by a group of authors from a single country, the number of multi-country papers has also increased over the years.Over the years, some 80% of papers were written by authors from a single country, while some 15.7% of papers included authors from multiple countries.Roughly 4.2% of papers could not be indexed with regard to author country.
Most computing education conferences change their location each year, but Koli Calling is stationary: it is always arranged in the Koli National Park in Finland (except for one year when the hotel was being renovated and the 2020 allvirtual conference due to COVID-19).The unique location in the middle of wilderness and the unique atmosphere and community have guaranteed that Koli Calling continues to be an attractive event and publication venue for international participants.Indeed, for many, it has become hard to resist the call to Koli.

IX. KEYWORDS AND THEMES
The trends, popularity, rise and fall of topics in the Koli Calling conference during its 20 years of operation are revealed by analysis of keywords.Fig. 6 shows yearly occurrences for those 20 keywords that have appeared in the top five keywords during one or more years of Koli Calling.The keyword K-12 has become popular only in the recent years.The keyword learning analytics started to become common after so emergence of learning analytics and educational data mining as research tracks started to attract computing education researchers, too (e.g.[42]).CS1, and other programming-related keywords have always been top keywords in Koli Calling, which is typical in other computing education conferences as well [5].When interpreting the findings, it is good to note that keywords are not always used consistently by authors in computing education research [25].
The network in Fig. 7 shows, which keywords are most commonly found together, and identifies clusters of keywords.The pink cluster centers around topics of diversity and educational psychology, such as gender, mindset and intrinsic motivation, with connections to teaching topics such as OOP and Parson's problems.The dark green cluster centers around K-12 computing and computational thinking with a strong flavour in programming including programming languages Python and Java.The light green cluster centers around CS1 and novice programmers and related research approaches such as cognitive load theory, phenomenography, constructivism, and visualization.The blue cluster is centered around automated assessment, algorithms, and project-based learning, while the yellow cluster centers around educational data mining, massive open online courses, databases and replication.The strong emphasis on programming, which is seen in both green clusters, is remarkable.However, similar observations about the dominance of teaching programming have been done with regards to other publication venues of CER, too [5], [15].

A. AUTHORS AND AUTHOR NETWORKS
Our first research question asked: ''How have authors and author networks shaped Koli Calling and its community over time?''Many of the most productive authors over the first 20 years have been from Finland, which is not surprising with Koli Calling having traditionally been a major venue for their community.There is also an increasing group of non-Finnish authors who have found a ''home'' in the conference; this trend can be seen in their sustained participation in Koli (see Fig. 2).These key persons have provided a stability for the conference as it has evolved from a local Finnish conference to a truly international conference.In particular the shift towards a more international conference steering committee and selecting Conference Chairs from outside the Finnish community have helped to connect the conference to the Swedish and Australian/New Zealand computing education communities, as well as the USA through the so-called ''Sweden Group'' of researchers.This development matches a development of the CER discipline from being relatively new and fragmented when the Koli Calling conference was initiated to the more mature field it is today [5].This is much due to the instrumental people running and developing the Koli Calling Conference, also being central in the development of the CER discipline.
The co-author network in Fig. 3 is another example of the internationalization of the conference and where the importance of some key persons become apparent.The importance of key people in bringing authors from many different countries is impressive.Many of the top 20 authors are also part of co-author networks, which is part of how they have contributed to the shaping of the Koli Calling conference over the past decades.However, with regards to repeat authorship one must note that some 69.5% of authors appeared only once in Koli Calling, while for those who appeared twice or more, the mean number of publications is 4.05.Thus, while there is a strong core community of authors in Koli, a large portion of authors appear only once.

B. PUBLICATION PROFILE OF KOLI CALLING
Our second research question asked: ''How has the publication profile of Koli Calling evolved in terms of most-cited papers, keyword trends and keyword clusters?''The analysis of keywords reflect that the conference is rather small and has had a clear focus on computing education and particularly aspects of programming and tools that can be used in computing education.There are some new concepts that have emerged, for instance computational thinking, learning analytics and K-12 education, due to the development of the CER discipline.The changes in topics seem to follow a general development of computing education, but still maintaining a relative clear focus.Two keyword-clusters formed around programming-related topics, with twists towards e.g.visualization, and learning designs such as flipped learning, while emerging topics such as learning analytics and gamification formed another cluster.Other clusters centered around educational psychology, society, as well as diverse topics and research approaches.Introduction of paper types, like system papers and theoretical research papers, and call for tools have contributed to the focus of the conference.
A strong focus in programming has been observed in many other venues of CER [5], too, such as in SIGCSE Technical Symposium, shown by an analysis conducted already in 2004 [10].Programming education has been found to be a central topic in many publication outlets of CER [5], [15], [16].The strong focus on rule-based programming can be debated.While the dependence on computational devices increases [43], and more people are affected by machine learning (ML) systems [44], this development could reflect also in more CER conducted on related topics.It has been observed that K-12 education mostly still focuses on teaching programming or the use of computer applications [44]- [46].Also, voices are calling for increasing the understanding of communities and their needs in technology development; sensing human networks and interactions, habits, behaviour, and culture, with increased role of design over programming skills, increasing the need to teach design research [45], [47], [48], another theme that is not very much seen in the analysis of keyword trends of Koli Calling.Learning process of ML differs fundamentally to that of learning data structures and algorithms.ML models are composites of parameters rather than human-readable algorithms [49].Rather than a logical proof, their verification process is a statistical argument, and learning ML requires efforts in collecting data, cleaning data, selecting a model, and statistical testing [49].The relevance of ML in CER is increasing, with emerging research arising here and there (e.g.[44], [49]- [51]), including new HCI (Human-Computer Interaction) courses with a focus on how ML applications are created [49].Focus on ML also increases relevance of training in basic probability and statistics [49].While nearly the entirety of CER focuses on classical programming [5], these are important themes that CER conferences such as Koli Calling could increasingly pay attention to in the future.

C. INTERNATIONAL COLLABORATION
Our final research question asked: ''How has Koli Calling evolved from the viewpoint of international collaboration?''Interpreting the bibliometric data reveals a number of trends in the development of the Koli Calling conference series into an international publication venue of high standing in the ACM community.The trend towards developing the conference as an international venue for computing education reseach can be clearly seen in Fig. 4. The early years show a conference more or less for the Finnish Computing Education Research community that changed into a fairly international conference with a substantial Finnish core of authors after a few years.This trend has, in the last few years, developed further and the conference entered a phase in around 2013 in which the Finnish portion is no longer always the largest.
Yet, countries and whole continents are greatly underrepresented on the map of contributions to Koli.While Koli Calling was set up to shake up computing education, in Finland and beyond, it seems that only few papers have dealt with challenges of CER in developing countries.This was already observed in 2010 [52], and the situation has not changed that much.For example, contributions from Africa are not so often seen in Koli, even when the virtual format would have enabled online presentations.Dominance of high-income countries in CER publications have also been found in the many central CER venues [5], [22]- [24].Indeed, the CER community as a whole could discuss what to do to increase diversity and better serve all geographical areas.

D. LIMITATIONS
Scientometric analyses are not without limitations.First, while scientific databases, particularly Elsevier's Scopus and ACM Digital Library, are generally well maintained, and Scopus has better accuracy than e.g.WoS (Web of Science) [53], the databases are far from perfect.Many issues weaken the data, including: inconsistent and unstructured keywords, missing fields, missing or inconsistent recording of references and citation metrics, and flaws in article classifications as research articles, posters, editorials, or reviews.Other problems arise e.g. from authors who have changed their names.Even with comprehensive cleaning and manual checking, using algorithmic and manual methods, and automatic and manual detection of mistakes, detecting each and every flaw is not possible.However, the sample of articles is a representative, if not comprehensive, and exceptionally well cleaned.Another challenge arises from the quantitative nature of the scientometric method, which will leave analyses and their interpretation to be shallow without a narrative perspective.It is also important to keep in mind that scientometrics is, in certain aspects, limited compared to reviews and meta-reviews, but superior in other aspects [5].In order to interpret the findings, we have involved experts with decades of experience in CER.

E. FUTURE RESEARCH
There are several important possibilities for future studies.Firstly, the scope of analysis must be extended to include the other central dissemination venues in CER.Secondly, future scientometric analyses could zoom into specific topical areas, specific author communities, and specific geographical areas.Future research could also increasingly connect metrical analyses with in-depth qualitative perspectives, in order to reveal deep insights into the evolution of the scientific field of CER.While CER has always been heavily dominated by education of introductory courses, it would also be important to focus more on emerging and evolving themes.Thirdly, metrics analyses of the impact of CER could extend beyond scientometrics to investigate how CER is received in social media, blogs, news and media.

F. CONTRIBUTION TO CER
This research adds a significant contribution into understanding of the research themes and topics, authors, internalisation, and most cited papers in one central publication outlet of CER.The analysis has provided a historical analysis that allows positioning the conference among the other central publication outlets of CER.Koli Calling is one in a limited set of dedicated publication outlets of CER [5].In order to steer the development of CER, it is crucial to understand the status quo.While CER evolves from experience reports to methodologically rigorous empirical research [5], it is crucial to investigate the dissemination venues, as they have a significant impact e.g. to career advancement.The research gap of this article is the lack of systemic understanding about the publication trends of Koli Calling, and other similar publication outlets of CER.This study could benefit CER researchers in all stages of their careers, students, educators, and editors.Making this research was also a personal learning experience for the authors, and gave us new insights into the scientific field of CER.

XI. CONCLUSION
Koli Calling is a very special conference, not least due to its choice of a stable venue in a very remote place.Starting out as a local Finnish conference it has evolved into a gathering place for a unique international community; a community where most of the most cited authors in CER have contributed to the development of the conference, and published there at least once.Many of these authors are now regular contributors and attendees.The uniqueness of the conference and the existence of a stable backbone of people running the organization has created a well known brand in computing education circles.The size of the conference and the ample opportunities for interaction, both academic and personal, has led to the development of its reputation as an important conference.
The size and remoteness comes, however, also at price, and there are large areas in the world that are not represented.A unique feature has been a strong focus on programming and tools over the years, but new and broader areas in computing education have also been introduced, especially in the last decade.It is interesting for all organizers of CER events to ponder on future strategies for the development and sustainability of their conferences, not least in the wake of the Covid-19 pandemic.That is, how can CER conferences such as Koli Calling keep their uniqueness and community building aspects and at the same time becoming increasingly international?What will the long term impacts of hybridformat conferences be, and how do conferences of CER best serve their communities, while addressing issues of climate change and sustainability, in times of geopolitical crises?We hope this paper provides communities of CER with insights and analyses that will help keeping publication venues relevant, communities strong, and enhance their reputations even further.

FIGURE 1 .
FIGURE 1. Number of papers presented and published in Koli Calling conferences, divided in single-country papers and multi-country papers (authors affiliated with institutions from different countries).Gray color indicates unavailability of country metadata.

FIGURE 2 .
FIGURE 2. Top 20 Authors' Production Over Time in Koli Calling Proceedings (all data).TC =Total Citations.

FIGURE 3 .
FIGURE 3. Co-author network of Koli Calling authors with most collaborators by using fractional counting [27].Node size indicates the number of unique co-authors, edge thickness indicates amount of co-authorships, and colors indicate communities of researchers who frequently collaborate together, according to Louvain modularity algorithm.

FIGURE 4 .TABLE 1 .
FIGURE 4. Koli Calling's evolution towards an international venue: countries of author affiliations in published articles.

FIGURE 5 .
FIGURE 5. Distribution of Koli Calling articles across the globe based on first author's affiliation in each article.

TABLE 2 .
Twenty koli calling papers with most citations in scopus.