A Systematic Literature Review of Empirical Research on Epistemic Network Analysis in Education

Over the past decade, epistemic network analysis (ENA) has emerged as a quantitative ethnography tool for modeling discourse in different types of human behaviors. This article offers a comprehensive systematic review of ENA educational applications in empirical studies ( $\text{n}=76$ ) published between 2010 and 2021. We review the ENA methods that research has relied on, the use of educational theories, their method of application, comparisons across groups and the main findings. Our results show that ENA has helped visually model the coded interactions and illustrate the connection strength among elements of network models. The applications of ENA have expanded beyond discourse analysis to several new areas of inquiry such as modeling surveys, log files or game play. Most of the reviewed articles used ENA based on educational theories and frameworks ( $\text{n}=53$ , 69.7%), with one or more theories per article, while 23 articles (30.3%) did not report theoretical grounding. The implementation of ENA has enabled comparisons across groups and helped augment the insights of other methods such as process mining, however there is little evidence that studies have exploited the quantitative potential of ENA. Most of the reviewed studies used ENA on small sample size with manually coded interactions with few examples of large samples and automated coding.


I. INTRODUCTION
Epistemic network analysis (ENA) has emerged as a method for discourse modeling. The method builds on the notion that ''the connections between ideas and actions are more significant to the learning process than either ideas or actions separately'' [1]. ENA was developed to make sense of such connections using a repertoire of network-based methods that include visualizations and statistical modeling [2]. The network visualization of ENA models the co-occurrence of, for example, codes in discourse, activities in log files, or elements of interaction in a chat [3]. Whereas the method has been conceptualized in education research, it has been used in a wide variety of research questions and applications [4]. Recently, ENA became part of a growing new community of quantitative ethnography that extends to different types of human behavior and applications [4]. Shaffer [5] defines quantitative ethnography (QE) as a strategy that integrates The associate editor coordinating the review of this manuscript and approving it for publication was Mehdi Hosseinzadeh . statistical inference with the interpretive capability of qualitative, grounded analysis. In addition to ENA, other methods were developed, such as Shaffer's rho for measuring the inter-rater reliability to improve the level of calculation of agreement between data coders [6]. Moreover, to facilitate coding, the nCoder tool was developed [7]. The package was recently updated to ncoder+ with a semantic add-on to solve the low recall of nCoder [8]. Similarly, the Reproducible Open Coding Kit (ROCK) tool was developed to ease human coding [9].
Shaffer et al. [10] argued that although network analysis offers an alternative to traditional statistical methods for modeling collaborative interactions, many network analyses illustrate the nodes' connections of large networks using summary statistics. ENA can help solve this shortcoming by offering an -arguably -better visualization that better summarizes large numbers of nodes in a network. The authors also argued that with traditional network analysis, it is difficult to visually compare two networks if the nodes and edges are not in the same location in a visualization and, therefore, ENA could offer comparable layouts for networks [11]. ENA was then developed to address these issues by using several mathematical principles that aim to quantify the strength of connections and offer fixed layouts as well as several options for comparing networks across groups [2]. Another feature of ENA was its emphasis on modeling dynamic interactions, which showed how and when different codes were shared among collaborators [10]. Due to the flexibility of the method and the presence of free and simple tools, ENA has since been used to model a wide array of topics and problems across several fields [12]- [14].
Recently, the presence of vast amounts of quantitative and qualitative data about learners and their behavior kindled the interest in exploring ENA in diverse applications. For instance, ENA has been used to explore learners' collaborative interactions and their engagement in different learning activities [15]- [19]. Other common uses of ENA include the exploration of professional epistemic frame development among students [12]- [14], [20]- [22]. ENA has also been explored as a method to predict learning performance [23]. Comparison across groups is a common technique to determine how groups differ in their collaboration, thinking, and strategies or approach to learning [24]. Many studies have compared student activities, such as students' collaboration using chat data [15], interactions using online discussions [23], performance in online assignments [25] and students' progress [26].
A scoping review released in 2021 offered a brief overview of papers that studied quantitative ethnography (QE) [4]. Whereas the scoping review addressed the main threads of research, several issues require examination regarding the analysis of research methods, for example, the application of ENA in different educational levels and specialties, the use of educational theories, comparison across groups, and the implication of ENA use in interpretating educational context as well as research findings, for example, impact and contributions. This systematic literature review offers an indepth review of the topic addressed by the scoping review and addresses the shortcomings thereof. We then offer a muchneeded discussion about how ENA has fulfilled its promises. To that end, we present a systematic review that highlights the uses, methods, applications of ENA methods as well as the gaps. The article addresses these issues by answering the following research questions.
• What ENA methods have been used to address educational applications and how?
• What are the main findings [results] from research studies that have employed ENA methods in education?

II. BACKGROUND
Epistemic network analysis (ENA) offers techniques for individual and collaborative contexts that use the so-called epistemic frames theory to model acting and thinking or the behavior of, for example, a community of practice (CoP) [27].
A CoP is a group of people who share a repertoire of knowledge and approaches to similar problems and goals [28]. Individuals reframe their identities and interests in connection to such communities as a result of participation in their practices. The identity of a CoP can be described as epistemic frames with five primary elements: skills, knowledge, identity, values, and epistemology [29]. ENA can model individual and group learning characteristics, such as action, communication, and cognition, by representing them as nodes in an epistemic network. The nodes are connected by edges, and the relative weighting of the edges reflects the strength of association between the nodes [10]. An important step in ENA is to code the dataset of discourse or activities. Coding is the process of bridging two worlds: the world of events and the world of interpretation by investigating how the codes from a Discourse (upper-case D for community discourse) are systematically related to one another in the discourse (lower-case d signifying a person or group of people) [30], [31]. For example, in an urban planner epistemic game, Nash and Shaffer [20] evaluated the extent to which students imitated their mentor's Discourse by determining whether they made the same connections as their mentors over time and if they could develop the ability to think like professional urban planners.
In ENA, the data are divided into segments -called stanza -based on the nature of the data and the research question. Elements within the same stanza are connected and linked together in the ENA model, whereas elements in the different stanzas are not. Chasler et al [32] showed that the co-occurrence of elements in a stanza is important for understanding the meaning of the discourse and offers a good approach to model the cognitive interactions. The main process of ENA starts by creating a matrix that represents the links between codes created by each data line. These matrices are summed to construct the network that is placed in space, where each dimension corresponds to the association between unique pairs of codes to represent the connectivity and strength of the codes. Then the network visualization is generated by aligning the projected points in space [10]. The position of the nodes and the centroid of the network are computed to generate network visualizations. The resulting ENA model contains information about (i) Codes (nodes), which are the people/ concepts connected in the ENA model, (ii) Relations (edges), which is how codes relate to each other, (iii) Stanzas, which are the units of identification based on either time or process, and finally (iv) Evidence, which verifies the connection between codes [33]. ENA can be performed using the web tool or the R package rENA [34]. It is beyond the scope of this review to offer a comprehensive overview of ENA, and readers interested in reading more about the theory and methods are advised to refer to the tutorial by Shaffer et al. [10] or Shaffer's textbook on quantitative ethnography [5]. For more about the mathematical foundations, readers are advised to read the work of Bowman et al. [2].

III. METHODOLOGY
The authors followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA 2020) which is a popular framework widely used across health, social and educational sciences for systematic reviews [35] and the eight essential steps of systematic review by Okoli [36]: (1) identify the purpose; (2) draft protocol; (3) apply practical screen; (4) conduct literature search; (5) extract data; (6) appraise quality; (7) synthesize studies; (8) write the review. The following section presents the main steps and how they were performed in the study.

A. IDENTIFY THE PURPOSE
The authors identified the lack of a comprehensive synthesis of ENA research and the clear need for a systematic review of epistemic network analysis in education based on research questions.

B. DRAFT PROTOCOL
After identifying the purpose and scope of the review, the second critical step prepares the protocol, which is a plan for the review steps to minimize researcher bias during study selection and data extraction [37]. Based on Fink [38], the authors frequently met to draft the protocol by writing down the strategy for conducting the review and practiced following the protocol to ensure complete reproducibility and consistency in the review execution. The protocol included generating the research question, the predefined strategy for the literature search, the search locations, the selection criteria, the assessment of studies, the data extraction strategy, and the planned timetable [37].

C. APPLY PRACTICAL SCREEN
The inclusion and exclusion criteria used in the selection of studies were based on the research questions and guided by previous reviews, for example, [39]. Articles that addressed an empirical ENA problem in education according to the following inclusion and exclusion criteria were included: 1. Articles are written in English. 2. The article is available as a full text and is peer reviewed. Thus, editorials, conference abstracts, and workshop proposals were excluded. 3. The research must be an empirical study that collects and analyzes empirical data with the appropriate methodology and results. Thus, reviews and theoretical or incomplete reports were excluded.

D. LITERATURE SEARCH
We identified five databases covering research at the intersection of social, educational and computer sciences relevant to our research questions: Scopus, Web of Knowledge, Springer, ERIC, and ACM Digital library as well as the first and second editions of international conferences of quantitative ethnography conferences (ICQE 2019 and ICQE 2020). The search formula was selected to cover all existing articles that are relevant to our research question. We used keywords with a wild card to capture all forms of the keyword. Thus, epistemic network * was selected to capture epistemic networks, epistemic networking, and epistemic network; quantitative ethnograph * was selected to capture quantitative ethnography and quantitative ethnographies. To capture keywords related to education, we used educat * to capture keywords based on the same stem, for example, education and educator. Similarly, learn (e.g., learning, learner), teach * (e.g., teaching, teacher), train * (e.g., learning, learner), collaborat * (e.g., collaboration, collaborative), cooperat * (e.g., cooperation, cooperative) and student * (e.g., student, students). Accordingly, the following search formula was used:

''epistemic network * '' OR ''quantitative ethnograph * '') AND (''educat * '' OR ''learn * '' OR ''teach * '' OR ''train * '' OR ''collaborat * '' OR ''cooperat * '' OR student * )
The search was conducted from 15 to 20 February 2021. The search yielded 395 articles from all selected databases (129 articles from Scopus, 48 articles from Web of Science, 176 articles from Springer, 23 articles from ERIC, and 19 articles from ACM digital library). All articles were imported into the Rayyan web-based system for analysis. Duplicates were removed, resulting in 291 articles. The abstracts, titles, and keywords of the first 100 manuscripts were independently scanned and reviewed by the first and second authors. The inter-rater agreement was 0.86, and manuscripts that had any conflict were discussed. The disagreements were resolved, and the first author proceeded with the filtering. The authors met to discuss and resolve uncertainties. The title and abstract scan resulted in 146 articles eligible for fulltext review, which resulted in 82 eligible studies based on inclusion and exclusion criteria.

E. DATA EXTRACTION
Data extraction is the process by which authors captured the key information and categories of the included studies in the form of a codebook. To increase efficiency, minimize individual variation between reviewers, and reduce error in data analysis, the study adopted a codebook from a previous coding scheme of Kaliisa et al. [4] for data extraction and categorization. Furthermore, the authors adopted other categories that related to educational research, such as theory background [28], [40], participants' educational level, coding [41], comparisons, outcomes, and implications for education [42]. Accordingly, the extracted data included (1) year and publication type, (2) sample and population categories, (3) the raw data source in each study, (4) comparisons between study groups, (4) type of coding in each study and its method of application, (5) the theoretical background and (6) the main findings of each article. The coding was initially performed by two coders independently for 10 studies. Thereafter, they discussed the coding challenges and finalized the codebook. One of the coders then continued with coding the articles and met with the second coder to discuss and resolve uncertainties.

F. QUALITY APPRAISAL
The extracted papers were examined more closely for quality. Four papers were excluded as they failed to meet the quality standards established by Fink [38] for presenting methodology, results, and conclusions. Two articles were excluded as they have the same sets of data published in a different journal and/ or conferences. Ultimately, 76 studies were included in the systematic review (Appendix). The flow diagram of the review process is shown in Figure 1.

G. SYNTHESIZE STUDIES
In this stage, the authors assembled, discussed, and analyzed the data to obtain a comprehensive sense of the collected data. The synthesis stage involves moving from an authorcentric to a concept-centric perspective by mapping all data evaluation and incorporating it into the review's hypothesis and structure [43].

IV. RESULTS
In this section we will present the descriptive analysis of the reviewed studies, study populations, source of raw data, comparisons included in the studies, coding, theoretical underpinnings and the main research findings in the reviewed studies.

A. DESCRIPTIVE ANALYSIS OF THE REVIEWED STUDIES
The total number of studies was 76 ( Figure 2). Most studies were journal articles (n=43, 56.6%), and the rest were conference papers (n=33, 43.4%). The studies were published between 2010 and 2021. The maximum number of studies in any year was 25 studies in 2019, when the first QE international conference was held in October 2019, followed by 2020 (21 studies) and 2021 (9 studies).

B. STUDY POPULATION
Most of the included studies were conducted on university students (n=39, 51.3%), followed by 23 articles (30.3%) on school students in all levels, five studies (6.6%) on teachers, postgraduate students in four articles (5.3%), three articles (3.9%) on simulator trainees as a part of continuous medical education, and three articles (3.9%) based on Massive Open Online Courses (MOOCs) (Figure 3). Most studies included a small sample size of less than 100 (n=62, 81.6%), 11 studies (14.5%) had 100-900 participants, and two studies (2.6%) had more than 1000 participants [23], [44]. Only one study did not report its study sample size [45].

E. CODING 1) CODING TYPE
The coding of the selected articles was deductive, inductive, or a combination of both ( Figure 6). The deductive coding was used in 32 (42.1%) articles with a top-down approach based on codes derived from previous research. Another 25 (32.9%) articles used inductive coding in a bottom-up approach that was generated from the data with no predefined coding framework using, for example, the grounded theory approach [21], [25], [45], [49], [51], [54], [60], [61], [64], [68], [69], [101], [102]. The third type of coding, which was reported in 19 articles (25%), used a combination of coding methods. Deductive codes were used first and were based on previous studies, followed by the inductive process of adding, extending, and refining codes.
Automatic coding was reported in eight studies (10.5%). For instance, LDAtopic modeling was used to automatically extract topics from the text [15], [23]. Similarly, automatic text coding was performed by identifying the word stem and its conjugations [25], and automatically generating log files of students' actions from learning management system [24], [86], [93], or educational games [89]. Finally, an algorithm was used to analyze eye-tracking during the solving of graph tasks by determining areas of interest (AOIs) [95].

F. THEORETICAL BACKGROUND IN SELECTED ARTICLES
Most of the reviewed articles use ENA based on educational theories and frameworks (n=53, 69.7%), with one or more theories per article, whereas 23 articles (30.3%) did not report theoretical grounding (Figure 9).

2) COLLABORATIVE LEARNING (CL)
Johnson and Johnson [105] define collaborative learning as a ''set of teaching and learning strategies promoting student collaboration in small groups in order to optimize their own and each other's learning''. ENA was used to study students' collaboration with each other and their mentors to frame, examine, and solve complex problems in different settings, for example, collaborative problem-solving (CPS) [71], [73], [75] communities of inquiry [48], computer supportive collaborative learning (CSCL) [49], project-based engineering [19] and scientific reasoning processes [16].

3) SELF-REGULATED LEARNING THEORY (SRL)
An oft-cited definition by Panadero [106] describes SRL as ''a core conceptual framework to understand the cognitive, VOLUME 10, 2022 metacognitive, behavior, motivational, and emotional aspects of learning.'' Learners effectively control their learning through internal and external feedback cycles of planning, performance, and reflection to control metacognitive and motivational behavior toward their goals in SRL. In the context of SRL, ENA was used in seven articles (9.2%) to analyze sequences of SRL processes [19], [93], identification of SRL in learning strategies [24], metacognition reflections [99], self-regulated behavior [86], feedback perception to adapt SRL processes [66] and finally, students' views in curriculum based on SRL [87].

4) COMMUNITY OF INQUIRY (CoI)
Garrison et al. [107] describe CoI as ''a pedagogical framework to examine the development of learning and cognition in online environments through three dimensional relationships called presences.'' ENA was used to analyze the relationship between one or more dimensions of CoI in seven articles (9.2%) [48], [55]- [59], [101].

5) PROJECTIVE REFLECTION (PR)
Foster et al. [79] define PR as ''a methodological and theoretical framework that considers learning as an exploration of identity.'' PR focuses on the integration between (I)dentity in a specific community of practice and self (i)dentity which reflects the personal goals. ENA explored identity in five articles (6.6%) through theoretical constructs of ''Knowledge, Interests/ Valuing, patterns of Self-organization/ Self-control, and Self-perceptions/Self-definitions'' [79], [84], [90]- [92].

7) NATURE OF SCIENCE (NOS)
NOS is the main focus of science education and a key element of science literacy, expressing a way of knowing that integrates the features of scientific knowledge development [109]. The connections between NOS aspects were explored using ENA in two articles [81], [87] and views of the NOS surveys were used in a one article [85].

8) COGNITION THEORIES
Metacognition -or being aware of one's learning and improving the learning experience using one's own cognitive resources [110] -was used in two articles that focused on the metacognitive components of knowledge, goals, and actions [68], [99], whereas one article was based on distributed cognition theory, which argues that cognition and knowledge are distributed among individuals and tools in the environment and are not confined to an individual [76].

9) COMPUTATIONAL THINKING (CT)
Wing [111] defines CT as ''a fundamental skill for solving problems, designing systems, and understanding human behavior that draws on concepts to computer science.'' The students' computational thinking was measured and analyzed using ENA in two articles [74], [82].

10) MICRO-TEACHING
Micro-teaching is an implementation method of dialogic pedagogical teaching. Student teachers are usually trained to play the roles of both teachers and students during practice microteaching. The teacher's role allows students to improve their teaching skills while the student's role helps them understand the psychology of the student [72], [103].
Other theoretical frameworks were reported in the reviewed studies, namely, knowledge building [83], the control-value theory of achievement emotion [66], and Language processing theory [47]. Finally, the achievement goal theory framework was reported [67].

G. THE MAIN RESEARCH FINDINGS IN THE ARTICLES
The most frequent theme in the research findings in the articles was related to learning interaction, which can be classified into learner-instructor, learner-contents, or learnerlearner interactions [42]. ENA was used to evaluatelearnerinstructor interaction [63]. Although the teacher's communication with students might show initial resistance to the mentor's frame, it subsides with mentor facilitation to develop a more professional frame [60], [70]. When studying discourse, the authors reported that virtual mentoring is as effective as face-to-face mentoring [63]. The study of the participant and pedagogy reflections revealed that ''mentor-reflectingon-student-action'' was related to ill-formed activity, whereas ''student-reflecting-on-mentor-action'' was related to more well-formed tasks [61].

1) LEARNER-CONTENTS INTERACTION
The learners' interactions with different discussion materials using ENA showed a more harmonious communication when students used interactive learning materials [18]. ENA analyzed the connections among the elements of professional skills and identity, for example, journalism [64], engineering [21], and urban planning [84]. Moreover, ENA was used to describe students' thinking in problem solving in a professional manner [64], the development of students' identity [91] and the stronger connections among the discourse elements of high performer students [78].

2) LEARNER-LEARNER INTERACTION (ANALYZING COLLABORATION)
ENA provided insight into CoI in online discussions for cognitive presence [55] and social relationships [57]. The temporal context of students' collaboration was explored in engineering design [61] and suggested that students with a more social exchange are more engaged in the planning and solving of collaborative tasks [19]. Similarly, ENA was used to investigate socio-cognitive activities and students' collaboration in problem solving [71]. In general, high performers had strong connections between adding concepts and creating links in the problem-solving process, which significantly differed from low performers' behavior [86]. Other analytical methods like process mining (PM) have been used with ENA to augment the insights [75], [88].

3) STUDENT EVALUATIONS
ENA was used to reveal the differences between high performers and low performers by modeling the connections between verbal codes [76], examining the connections between skills and decision-making [46], and evaluating the scientific reasoning [16]. The high-performing group create an actual open-ended learning experience and develop higherorder thinking whereas the low-performing group focus on the link between knowledge and learning activities [50]. Although ENA can function as an assessment tool for teachers to assess assignments and interpret their contents, Fougt et al. [25] reported that they were unable identify significant differences of different performance levels in assignments.

4) NATURE OF SCIENCE CONNECTIONS
ENA was used to explore learners' connections among NOS elements to identify the quality of students' understanding of NOS [81], [85]. Thus, ENA was shown to be useful for exploring pre-conceptions and post-conceptions by identifying the presence of ideas and the change of ideas [87].

V. DISCUSSION
ENA was conceptualized in 2009 to offer a quantitative method for studying coded discourse [3]. During the last three years, ENA witnessed increasing interest in the methods, a growing community, a yearly scientific event, and an expanding miscellany of applications that extended to other fields in addition to education. The ENA toolset has also expanded to include web applications and R packages, for example, rENA [34] and ncodeR [7]. The current pace of growth suggests that ENA adoption will result in more and diverse scientific output. Therefore, our systematic review sought to analyze the emerging field to highlight current achievements and future opportunities. Our study analyzed 76 empirical studies published between 2010 and 2021, and 2019 had the most papers. In the last two years, 30 papers were published.
Sixty-two studies (82%) in our review analyzed samples of less than 100 students, with a median of 32 students. Fourteen studies (18%) analyzed more than 100 students, and only two studies had more than 1000 students [23], [44]. Whereas this limited number of study participants could be explained -at least partially -by the need to code the data, it indicates that ENA has been a tool for research rather than practice with limited scalability. Recently, automatic coding using, for example, topic modeling [15], [23] has emerged, as well as ''semi-automated coding'' where part of the data is coded manually (as a training dataset) and the remainder is automatically coded based on the training data [47]. Progress toward automatic coding would accelerate the uptake of ENA as a scalable tool that can be embedded in, for example, dashboards to help teachers or students.
Many of the studies have analyzed online discussions or interactions among learners, and they constitute almost half of the studies in our review. Interestingly, the other half used data that was not related to students' interactions or conversations (interviews, assessment tools, log files, and observations), which signals the expanding repertoire of the applications of ENA beyond discourse analysis and to new areas of inquiry. Theories related to collaborative learning (CoP, CL, CoI) constituted almost half of the included studies, and SRL -as a theoretical underpinning -came a distant second with around 10%.The remainder consisted of other theoretical frameworks (e.g., projective reflection, TPACK, and computational thinking) or no clear theoretical framework in 23 studies (30%). Such a picture reflects the interest in exploring the relational insights of ENA to model the relations and the structure of connections between the studied elements in new areas.
The prevalence of diverse types of data, inductive coding and the use of various theoretical frameworks point to many papers that tried ENA to solve existing problems, shed light on new aspects, or explore new areas of inquiry. Similarly, several studies have explored ENA as a complementary method to existing approaches. For instance, [23] explored the value of combining SNA and ENA in what they referred to as a social epistemic network signature (SENS) to predict students' performance. Similar combinations with SNA have also been explored to examine teachers' agency [94] and participation in knowledge building [80], [83]. Recently, ENA was used in tandem with process and sequence mining to augment the derived insights and reveal the strength and magnitude of connections between students' SRL elements [24], [88], [93].
Perhaps the most important objective of this study is how ENA has contributed to our research and practice. The largest number of the studies used ENA to evaluate, understand, or compare aspects related to students' interactions with each other, with content, or with teachers. Studies evaluating students and mentor interactions reported that ENA helped to assess mentoring and examine teacher facilitation and learners' reflection on teacher mentoring [12], [63]. Similarly, several researchers have reported the utility of ENA to evaluate the development of professional abilities in journalism, engineering, urban planning, script writing, and surgery, for example [21], [52], [64], [77], [84] The largest group in the studies have assessed collaborative settings. The most common findings relate to revealing the types and strengths of presence in CoI [101], assessing patterns of knowledge exchange in collaborative learning [80], [83], or understanding problem solving behavior [71]. Less commonly, ENA was utilized to assess the nature of scientific thinking [81], computational thinking [82] VOLUME 10,2022 and TPACK [26]. A small subset of the studies explored students' perception and sense-making of feedback in terms of value, impact, and quality [67]. Comparing and contrasting high and low achievers to infer differences and similarities has also been a common theme for evaluating differences in verbal discourse [76], scientific reasoning [16] and higherorder thinking [50]. However, studies assessing the utility of ENA as an assessment tool were unable to identify significant performance differences in assignments [25].
The studies in our review included a type of comparison, and the most common comparisons (23) addressed differences in performance (30%), 19 examined differences in temporal progress (25%), and 10 focused on differences between participant subgroups (13%). However, this comparison has been performed visually in many studies (43%). Studies comparing the differences among groups have mostly used pairwise comparison along the X-axis or Y-axis. A drawback of such comparison is the difficulty of communicating the results in an easy-to-understand way or translating differences along coordinates into interpretable conclusions. Furthermore, most of the existing comparisons have not compared across several groups (more than two), and they use prepost design or report effect sizes to quantify the magnitude of such differences with effect size.

5) HAVE ENA FULFILLED ITS PROMISES?
We offer a concise overview based on the literature covered in this study and the seminal papers by the founders of the field to answer the question of how ENA has realized the aspirations behind the establishment of the field. One of the main promises of ENA was to offer quantitative analysis of coded data, which Shaffer et al. [10] stated as follows: ''ENA identifies and measures connections among elements . . . and measures the strength of association among elements in a network.'' However, the reviewed studies have barely reported the quantified structure of discourse or the strength of association between codes. Similarly, ENA was proposed as a method to ''assess learner performance.'' However, the reviewed research has reported aggregated results of groups in most studies, with just one study in which the authors concluded that using ENA did not help assess students' performance [25]. ENA was built with temporality in mind. According to Shaffer et al. [10], ''using ENA to create a trajectory model, which indicates changes in structures of connections over time.'' We have not seen such trajectories in the reviewed papers, and the papers that examined progression have compared different aggregated networks of certain periods -an approach that strips the network of its longitudinal aspect. Whereas many studies harness the co-temporal nature of interactions, these papers display the results in aggregated networks and, therefore, the aspect of temporality is lost.
Furthermore, ENA promised to enable ''the analysis of networks too large for multivariate parametric techniques'' [10]. However, the majority of the networks in our study were small with a limited number of nodes (median =8). Studies that used ENA with a larger number of nodes were difficult to interpret or visualize with several overlapping edges and nodes [81], [87]. Another promise behind ENA was that the ''network graphs allow us to interpret the significance of the locations of the points in the ENA model'' [10]. This promise was achieved to a degree in the reviewed papers, as visual comparisons were prevalent in all our studies and easily showed the networks and their connections. However, it was unclear how the reported results by the reviewed papers help explain such differences to the reader or practitioner. In other words, if a study found a significant difference on the Y-axis between group A more than group B, how can these results be communicated simply and clearly to practitioners? Moreover, how are such differences translated into practice?
The literature review has left us pondering the comparison between ENA and SNA and multivariate analysis which are methods that ENA was built to address some of their shortcomings [2], [10], [112]. SNA offers several advantages for quantifying the overall networks (e.g., density, reciprocity, and efficiency), node positions (e.g., centrality measures), and connection strength (e.g., edge weights and edge centralities) [113], [114]. Such quantitative analysis has been immensely useful across vast fields of research [115]. ENA methods have no comparable quantitative measures and researchers cannot, for instance, report which was the most central node that bridges others or the node that was close to other codes. SNA offers several null models that allow researchers to compare networks under study to random network models to determine whether their networks are statistically meaningful [116]. Similarly, SNA offers several inferential network methods and robust confirmatory tests that help understand why edges form and, thus, help researchers build or contribute to hypotheses [116]. SNA has a vast community with several threads of research, a large repertoire of methods, open software tools, and solid theoretical foundations. However, ENA is maintained by a small group of researchers concentrated in a single institution with limited contribution from the wider community to the theory, statistical foundations, or development. In comparison, the network psychometric field has emerged to analyze multivariate networks, and it has a large community that contributes to the tools, methods, and theory, offers several network confirmatory tests, for example, network bootstrapping, and it has vibrant discussions regarding the theoretical and mathematical foundations of the methods [117]. It is unclear whether ENA has addressed the said shortcomings of the two methods. ENA may have offered alternative methods for implementing or improving some existing functions currently offered by network analysis methods. However, these alternative functions came at the expense of losing access to a wealth of potentials offered by the network ecosystem shared across several fields, for example, network science and SNA. The lack of ENA networks' inter-operability with existing methods or an export file format makes integration with other tools impossible.

VI. CONCLUSION
In summary, a large volume of research has explored the potential of ENA across a diverse range of problems. The current analysis supports the conclusion that ENA has helped visualize connections between coded elements, enabled comparisons across groups, and helped augment the insights of other methods, for example, process mining. However, ENA has not been able to fulfill many of is aspirations. The current implementations of ENA have neither been scalable nor automated. Our analysis has also shown that there is insufficient evidence that ENA has helped quantitatively investigate qualitative data nor has it helped assess learners' performance nor chart the temporal trajectory of interactions and therefore, it is fair to conclude that the expectations for ENA were set too high, but many fell short of promise. The small group of developers behind the development of the theory, software, and conceptualization of the field limits the role of the broader community to mere consumers of locked tools rather than partners who can contribute and drive the field forward. Whereas we expect more growth in applications in both volume and breadth, we hope for more involvement of the wider community in shaping the perspectives of the methods. Only then can we expect more diversity in implementations, richer quantitative capabilities, and novel perspectives. Table 1.

See
RAMY ELMOAZEN is currently pursuing the Ph.D. degree in learning analytics with the University of Eastern Finland. He is also a Researcher at the School of Computing, University of Eastern Finland. His research interests include developing computer-supported collaborative learning and learning analytics, including social networks and epistemic network analyses.
MOHAMMED SAQR received the Ph.D. degree in learning analytics from Stockholm University, Sweden. He had a postdoctoral training at the Université de Paris, France. He currently works as a Senior Researcher at the University of Eastern Finland, Finland. He is working on artificial intelligence, big data, network science, and scientometrics. His research interests include advancing novel methods, such as temporal networks, graphical Gaussian models, multichannel sequence analysis, and temporal processes in general. He is also an active member of several scientific organizations and acts as an academic editor in leading academic publications.
MATTI TEDRE is currently a Professor of computer science, especially computing education, and the philosophy of computer science, with the University of Eastern Finland. His recent book Computational Thinking (The MIT Press) with Peter J. Denning presented a rich picture of computing's disciplinary ways of thinking and practicing, and his 2014 book Science of Computing (Taylor & Francis/CRC Press) portrayed the conceptual and technical development of computing as a discipline. He has given more than a dozen conference keynotes on computing education, including ACM ITiCSE, ISSEP, and ECSS.
LAURA HIRSTO is currently a Professor of educational science with the Department of Applied Educational Science and Teacher Education, University of Eastern Finland. She is also supervising various research and development projects related to academic and educational development, student learning and teacher learning, and pedagogical perspectives on learning analytics in various contexts. Her research interests include higher education students' and teacher students' as well as primary level pupils' learning and motivational processes, and in variations of effective teaching and learning environments.