A Deep Learning Model to Predict Student Learning Outcomes in LMS Using CNN and LSTM

Learning Management Systems (LMSs) are increasingly utilized for the administration, tracking, and reporting of educational activities. One such widely used LMS in higher education institutions around the world is Blackboard. This is due to its capabilities of aligning items of learning content, student-student and student-teacher interactions, and assessment tasks to specified goals and student learning outcomes. This study aimed to determine how certain Key Performance Indicators (KPIs) based on student interactions with Blackboard helped to forecast the learning outcomes of students. A mixed-methods study design was used which included analysis of four deep learning models for predicting student performance. Data were collected from reports on seven general preparation courses. They were analyzed using a documentary analysis approach to establish possible predictive KPIs associated with the electronic Blackboard report. Correlational analyses were performed to examine the extent to which these factors are linearly correlated with the performance indicators of students. Results indicated that a predictive model which combined convolutional neural networks and long short-term memory (CNN-LSTM) was the optimal method among the four models tested. The main conclusion drawn from this finding is that the combined CNN-LSTM approach may lead to interventions that optimize and expand use of the Blackboard LMS in universities.


I. INTRODUCTION
A Learning Management System (LMS) is an application of purposefully selected software that supports the learning process in higher education institutions. It acts as an automated system for the administration, tracking, and reporting of educational activities and learning outcomes [1]. LMSs are designed and implemented to help streamline the education process -including teaching, learning and administrationthrough the identification and assessment of students and The associate editor coordinating the review of this manuscript and approving it for publication was Seifedine Kadry . institutional learning goals, by tracking progress towards meeting the set goals, and the collection and presentation of data for supervision of the learning process. In this way, an LMS is not only useful for delivering learning content but also for the management of student uptake and compliance, and the analysis of knowledge and skills gaps [2].
One widely used LMS in higher education institutions around the world is Blackboard. This is a technology platform utilized by educational institutions to support the exchange of important learning content and materials, student assignments and reports, and announcements by the teacher. Additionally, Blackboard technology facilitates real-time activities VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ including discussion forums and chat rooms for studentstudent online interactions and for the transfer of documents, resources, and questions between students and teachers [3]. With advances in technology, LMSs are increasingly utilized to forecast student learning outcomes as well as to monitor student performance. The affordances of Blackboard mean that all content, interactions, and assessment tasks items can be aligned to specified goals or student learning outcomes. Alignment to activities for assessing students, including assignments, examinations, and discussion forums, allows for the collection of student performance data at the course level to establish the extent to which students are achieving the stated goals. Students can also keep track of how their learning is progressing in relation to those goals. Further, such systems generate and store large amounts of meta-data. These include but are not limited to: number of hits to the course content, number of visits to the course, duration of interactions using the online systems, number of downloads, and date of visit. This data can be utilized to generate valuable information to assist staff with the decisionmaking process [4], [5].
The aim of paper is to predict student learning outcomes and to monitor their performance during the educational process. The method utilizes automatically generated data from the online LMS. More specifically, we examine how seven selected Key Performance Indicators (KPIs) in Blackboard help educators to predict the learning outcomes of students. In order to achieve this goal, there is a need to apply artificial intelligence models. Artificial intelligence is referred to as to the general capacity of computers to imitate human thought and perform tasks in real-world setting, whereas machine learning refers to the types of technologies and algorithms which support systems to identify patterns, engage in decision making, and generate improvement through experience [6]. Deep Learning is generally described as a form of machine learning based on learning data representations. In this field, machine learning refers a form of artificial intelligence that includes systems which can learn from data, identify patterns, and implement decisions with minimal human intervention [7].
Deep learning permits computational models made up of multiple processing layers to 'learn' representations of data with several levels of abstraction. Deep learning is associated with significant advances in problem solving which have not been fully resolved by the artificial intelligence community. This is because it is effective at discovering intricate structures within large data sets through the use of a backpropagation algorithm [6], [7]. The algorithm is used to determine how the machine should alter its internal parameters being utilized to compute the representation in each layer from the representation in the layer before it. The deep learning process therefore involves each new level 'learning' to transform the input data into a slightly more composite and abstract representation [6], [7].
In this paper, a new deep learning model using convolutional neural networks (CNN) and long short-term memory (LSTM) is developed to predict student performances. It provides valuable information for universities which can help to ensure the quality of their services. It can also help with the development of strategies as well as ensuring student success by providing them with tailored support based on their predicted performance.
In summary, the main contributions of this paper are presented as follows: 1) Analysis of the level of support provided to universities to exploit student meta-data generated from the online LMS. 2) Analysis of deep learning models for predicting student performance. 3) Analysis of the correlation and time series of student performance at university per attended course. 4) Comparison of the results obtained using CNN-LSTM with CNN, recurrent neural networks (RNN), LSTM, and CNN-RNN for the prediction of student performance.
The remainder of this paper is organized as follows: Section II presents a review of relevant literature. The proposed method is presented in Section III, followed by details of the experiment results and discussions in Section IV and V. Finally, Section VI presents a conclusion in the paper.

II. LITERATURE REVIEW
Most universities now utilize technology platforms to improve education service delivery. This highlights the importance of activating technology in response to changes in students' learning needs and education provision more broadly. Indeed, building a big institutional dataset of student activities is important for monitoring student retention, progression, and attainment, and can be accomplished through a LMS [8]. This literature review is divided into two parts: a) use of LMSs in education; and b) predicting student learning performance using LMSs.

A. THE USE OF LMS IN EDUCATION
Various LMSs may be adopted to facilitate educational processes such as Blackboard, Desire2Learn, Moodle, and Canvas [9]. These LMSs share similar features in that they provide asynchronous interaction, at any time, in any place. In addition, they incorporate services such as collaborative learning, video conferencing, file sharing, student grading, and a discussion forum. Significant differences among the systems are rare, however, due to the high level of competition between the systems commercially. Yet, while differences in the features of the systems are not salient, differences in usage are prevalent geographically. Table 1 shows the general global use of various LMSs across the world. Use of Blackboard in the United States (US) and Canada is at 33%, whereas use of Moodle and Canvas are both at 20%. By contrast, use of Moodle in Europe is at 65% compared to Blackboard (12%). In Saudi Arabia, most public universities (25 out of 28) use Blackboard over other systems. According to a study conducted at the University of Ha'il, the Blackboard communication platform offers effective communication and content sharing features [10]. These include announcements shared by faculty members with students, chat functions, discussions, email capabilities, content sharing, a calendar, assignments, a media library, and assessments. However, studies have also identified some issues that may interfere with the ability of students and instructors to collaborate effectively over the Blackboard platform [11], [12].
Additionally, faculty members must be motivated to become proficient users of Blackboard or else they miss the benefits of its innovative pedagogies. [13] support this result, reporting that 80% of students felt Blackboard was convenient to use in learning. In addition, 34% of students at the University of Ha'il felt Blackboard improved their learning experience, and 41% of students felt it helped to improve student-instructor communication. However, these results were obtained before the University of Ha'il had implemented the use of Blackboard for all courses, as was the case during the University lockdown due to the COVID-19 pandemic. According to students at the University, Blackboard was incorporated into classroom learning systems for all courses. Moreover, faculty members continue to be encouraged to use it to support curriculum implementation.
It is noted that Blackboard creates customized course management through its software, or Building Blocks, which use open application programming interfaces (APIs) and web services produced by third-party developers. However, Blackboard does not provide a comprehensive and professional system for analyzing student performance to assist decision makers to forecast activities that best promote the achievement of program learning outcomes (PLOs). In response to this issue, we adapted effective strategies and KPIs which may assist students to improve their achievement of the PLOs.
In summary, Blackboard provides a virtual learning environment that enables easy communication between students and instructors. It also allows educational institutions to perform tasks virtually such as lectures, assignments submission, and exams. Conducting all learning tasks in one virtual environment provides auto-generated data which allows educational institutions to gain more insight into student performances, thus supporting their decision making. Further, Blackboard generates data on student interactions within the learning environment. This allows educational institutions to broaden the measurement of student performance as well as to analyze this data to customize academic support to students and improve their overall learning experience. However, autogenerated data can be difficult to analyze due to its large volume and variety. Therefore, in the next section we analyze the level of support provided by researchers to aid educational institutions to analyze auto-generated data from virtual learning environments [14].

B. ANALYSIS OF CURRENT STUDIES ON THE PREDICTION OF STUDENT LEARNING PERFORMANCE THROUGH LMSs
The use of LMSs to investigate student performance and to monitor students' learning progress has increased significantly. By applying such systems, we can potentially provide effective strategies and KPIs to help students to improve their PLOs. This section analyses the systems and methods identified in other studies which can be applied in Blackboard to achieve the goals of this research.
One technological strategy for monitoring student learning is use of the MSocial system. It is integrated into the LMS in order to monitor student activities in social networks [15]. Analysts apply this system to conduct a Social Network Analysis (SNA), which focuses on examining students' uses of social media as a part of learning. Specifically, researchers apply SNA to understand, visualize, and analyze students' social participation and interaction, and how these factors may enhance the learning process.
[16] expands the approach to monitoring student progress by considering the effect of social media on student learning and the need to identify the strategies, methods, and tools which assist researchers to analyze, report, and provide recommendations to improve student performance. Teachers can gain insights into the participation of a student through a display of activities and KPIs. However, the results reported by [16] do not include the educational context, overlooking the student's role in using e-learning tools and the culture of using such tools in the learning institution. Another issue with their method is that the use of social media is dependent on several variables such as student preferences and the possible negative or positive effects of social media on student education.
Other studies highlight the importance of considering the learning approach in relation to student KPIs [17], [18]. For instance, a learning approach refers to the bridge between learning environment and learning styles as influenced by a person's character [17]. [18] refer to a learning approach as the learning motivation by which suitable strategies in teaching are implemented. However, student KPIs do not always influence student performance and their final grades. As a result, the current study does not focus on the learning approach, and specifically the environment, when interpreting data related to student KPIs and their effect on student grades.
Another method of monitoring student performance is Learning Analytics (LA), which involves the use of large datasets to inform faculty members of students' learning and university experiences [19]. Many researchers have focused on applying LA in the educational context [20]- [23]. Moreover, LA is used to identify students who are at risk in order to support their learning progress [24]. However, the current study focuses on both diagnosing the issues students may encounter and on proposing educational practices that may assist them to achieve the intended learning outcomes at a high level.
[25] expand on the ideas of [24] with their suggestion that measuring learning experience effectiveness should be supported by LA through feedback on learning design (LD). [26] support this view by criticizing the focus on LA without sufficient consideration of the educational environment. For example, educators identify student activities through the number of clicks, essays, and discussion posts. However, LD can assist researchers to determine which variables tend to generalize the findings to various educational contexts, and how to action the findings [27].
Analysis of previous studies reveals a lack of research that predicts student outcomes based on their interaction with LMSs. They have focused on LA and the learning environment, while the application of a LMS to forecast KPIs related to the intended learning outcomes has not been fully covered. It appears that to assist decision-makers to match the university's general requirements with the activities which enhance the intended learning outcomes of students, a combination of Blackboard LA and KPIs would be useful [28], [29], [32]. Table 2 provides a summary of the literature review regarding methods previously employed to analyze student performance using LMSs.
It should be noted that the researchers in this study faced no issues regarding educational context when monitoring student KPIs because the context was almost identical for all students. This was for two reasons. First, all students shared the same requirement; that is, they take their courses using Blackboard, due to the University lockdown in response to the COVID-19 pandemic. Second, student conduct with regard to the investigated courses aligned with general university requirements for all students across the institution.
The researchers encountered no issues with regard to LD as the activities were provided in the course specifications and course reports. Thus, LA was the focus of the current study as the findings would be useful for identifying the educational activities and KPIs to enhance student achievement in relation to the intended learning outcomes [34]. Educators adopting this approach can predict issues to potentially affect student learning, revise the activities designed for students, forecast possible learning competencies, and implement effective activities in general courses. Based on these perspectives, the main research question is: How do the LA of KPIs in a LMS help educators to forecast the intended learning outcomes of students? The method proposed to answer this question is introduced in the section below.

III. PROPOSED METHOD
The design goal of the proposed method is to predict student learning outcomes and performance in LMS at a higher education institution using two methods: CNN [35] and LSTM [36], [37]. By combining two methods; namely, 1) CNN to extract effective features from the data, and 2) LSTM to identify the interdependence of data in time series data, the performance prediction accuracy was improved compared with state-of-the-art methods.
The prediction framework is shown in Fig. 1, and the main steps of the CNN and LSTM prediction model are introduced as follows: 1) Collect students' data from Blackboard, a common LMS used as a tool for storing university student data efficiently over time according to an analysis of predictions of student performance. 2) Select the significant features and eliminate abnormal values for each course to obtain valid data on students' performance as mentioned in Table 3 . Seven courses were selected and seven features for each course were analyzed.  3) Divide student data S into S Training and S Testing sets. The time series students' data is S t , the size of training set S Training is t, the size of the test set S Testing is V − t, where V is the size of students' features in S t 4) Extraction of students' features is applied by Feeding S train set into the convolutional layers and the max pooling layer of the CNN model. Input data is calculated as S conv = S train * K, S maxpool = Max(S conv ). Here, S conv is defined as the convolution layer result from trained data. The * represents convolution operation, and K is defined as the convolutional kernel, which is the convolution window size. S maxpool is the result of max pooling the layer of CNN. 5) Feed the extracted features of students S maxpool to the LSTM model. S maxpool is passed through three gates: input gate, forget gate and output gate of LSTM. 6) Prediction results on S Testing sets are gained by training and learning.

A. DATA COLLECTION
Students' performance dataset was acquired from the report of students' KPIs based on seven general preparation courses in Blackboard. The report is a combination of one report for all students, one for each student, and one for each course. These reports are provided by the IT Department at the targeted university which include electronic data associated with the Blackboard distance learning system. They outline the general university requirements for all undergraduates according to their specializations at the university. This comes in the form of students' cumulative data which include: 1) courses, 2) activities in course, 3) assessment methods, 4) grades, and 5) materials. We utilized the students' performance dataset because it purely contains features reflecting students' academic performances and online behaviors. The dataset consists of 35,000 student records with seven features mentioned in Table 3 . Each student studied seven courses related to four subjects: English, Mathematics, Physics, and Arabic language. As a result, the size of student record is 7 Courses × 7 Features = 49 per student which makes the total dataset size 35, 000 × 49 = 1, 715, 000. VOLUME 10, 2022

B. FEATURES SELECTION
Feature extraction imports student performance data from the Blackboard system and selects significant features to save into a feature vector. The features are described in Table 3 . A 1D vector of student features is considered as the KPIs data in this paper. The selected student features as shown in Fig. 2. are considered in the prediction CNN-LSTM problem as an input and sequence labelling model. Both the fetched student features and the output labels are constructed as sequences and saved into a feature vector. This is denoted by an input ,f7 ] with i = 1 to n = 7. These features were examined for various preparatory year courses studied during the first and second semesters.

C. FEATURES EXTRACTION USING CNN MODEL
A new deep learning model using CNN and LSTM was developed to predict student performance in each course.
In the proposed prediction model, CNN was used to extract the time series of student features and LSTM was used for performance prediction. This made full use of the time sequence of student data to obtain more reliable predictions. Second, by comparing the CNN-LSTM evaluation indexes with CNN [35], LSTM [36], RNN [40], and CNN-RNN [41], our method had good prediction accuracy and was better able to predict student performance within our higher education institution. The CNN model is a type of feedforward neural network developed by [38]. CNN presents good performance in many applications such as image recognition, healthcare analysis, and predictive analytics, and is one of the most well-known deep learning models. CNN can be effectively applied to the prediction of time series data and is composed primarily of two parts: the convolution layer and the max pooling layer. In our method, four convolution layers and one pooling layer were applied. Each convolution layer contains a plurality of convolution kernels and its equation after the convolution operation is: where l t represents the output value after convolution, tanh is the activation function, x t is the input vector, k t is the weight of the convolution kernel, and b t is the convolution kernel bias.

D. LSTM TRAINING AND PREDICTION MODEL
The LSTM model is an extension of the recurrent neural network developed by [42]. It is commonly utilized in text analysis and speech recognition. It includes a memory cell to support accurate predictions. Recently, it has also been adopted in the field of time series data prediction. The LSTM has three main components: 1) forget gate, 2) input gate, and 3) output gate as presented in Fig. 3. The LSTM calculation process is as follows: a) The forget gate considers the output value of the last cell and the input parameter of the current time as its input. The output value of the forget gate is calculated as follows: where the output is in limit of f t [0, 1], W f is the weighted valued of the forget gate, and b f is defined as the forget gate bias function. x t is considered as the input value of the current time, and h t−1 is the output value. b) h t−1 and x t are used as the input values for the input gate. The output and memory cell state of the input gate are calculated as follows: where i t (0, 1), W i is the weighted coefficient of the input gate, b i is computed as the input gate bias function, W c is the weight of the candidate input gate, and b c is the bias of the candidate input gate. c) Change the current cell state as follows: where the value range of Ct (0,1). d) The output h t−1 and input x t are received as input values of the output gate at time t, and the output O t of the output gate is defined as follows: Here, the o t (0, 1), W o is the weight of the output gate, and b o is defined as bias function of output gate. e) LSTM output value is using the output gate output and the state of the cell as follows: The training and prediction steps were introduced as follows: a) Initialize the CNN model training parameter: Wi is the weight coefficients and the deviation b i as shown in Table 4 . b) Prepare the training set S Train which includes the students' data in time series t. S train represents 70% of the dataset. c) Fed to the S Train as input layer, and passed to the CN layer to calculate S conv and then transferred to the max pooling layer as S maxpool according to Fig. 1. d) The effective features S maxpool are extracted and then fed to the LSTM model to produce the output result h t as shown in Fig. 3. e) The predicted valuesx i are then estimated based on the fully connected layer. f) To improve training in the model, data x i is normalized as there is a reasonable distance between student data in the input gate. The z-score method is applied to normalize the input data as follows: where y i is defined as the normalized value, x i is the input data for each student, and x is the average of the student performance. s is the standard deviation of x i . g) Error estimation: the estimated value is defined by the output gate andŷ i is compared with the observed value of this data group y i to estimate the corresponding error. h) Validate if the weights W i ≤ a specified criterion, predetermined number of epochs is defined that make the training model completed with the lowest error rate. Update the CNN-LSTM model and go to step 10; or else go to step 9. i) Propagate the calculated error in the backward way, change the weight and bias function for each layer, and go to step d to continue to train the model. j) Save the trained model for prediction. k) Set input testing S test with size 30% of dataset to predict their values. For more explanation, see the flowchart of the prediction process in Fig. 4.

IV. RESULTS
Codes in MATLAB R2020b were used to develop the CNN-LSTM algorithm. Data were exported from the Blackboard reports for each course. The MATLAB Coder, with a Deep Learning Toolbox, was applied to generate the C++ code containing the basic framework of multiple deep neural networks using a combined CNN-LSTM method. During the experimental process, 70% of student data were selected as the training set and the remaining 30% as the test set. As for the 1D CNN and LSTM models, the number of training epochs was 100 as mentioned in Table 4. During training, at the end of each epoch, the accuracy of the proposed 1D CNN and LSTM model with regard to the training and test datasets were estimated. This was done to help judge whether the model was overfitting and thereby to verify the generalization ability of the current model. The first experiment was conducted based on the proposed method and evaluated based on the evaluation metric: Accuracy = #No. of correctly predicted student data Total number of true students data . The accuracy of our CNN-LSTM method is presented in Fig. 5(a) and Accuracy vs. Loss results are presented in Fig. 5(b). While VOLUME 10, 2022 the number of epochs reached 100, the approximate accuracy of training data produces a reasonable result (96.2%), whereas the accuracy of testing data is 94.3%.

A. PERFORMANCE EVALUATION
Many factors affect the prediction model including: a) size of CNN convolution filter, b) neurons in LSTM, and c) LSTM batch size. We utilized three evaluation metrics to examine the prediction accuracy and its performance according to the above-mentioned factors: 1) Loss function (LOSS).
3) Mean absolute percentage error (MAPE). These metrics are defined as follows: where n is the number of input samples, y i is defined as the predictive value, and y i represents the observed value.
The main goal is for loss to have the lowest rate and the lowest running time cost.

1) SIZE OF CNN CONVOLUTION FILTER
A kernel can produce dimensionality reduction of the input data which can improve the learning rate of the CNN model to extract significant features. Results of the experiment presented in Table 5 show CNN filter size=10 provides the lowest loss value and low prediction errors (RMSE, MAPE) compared with other filter sizes.

2) NEURONS IN LSTM
LSTM neurons influence prediction accuracy. To determine the best number of neurons, we examined the proposed method according to various neurons [3, 6, 9, and 12] as presented in Fig. 6. We found the optimal number of LSTM neurons is 6.

3) LSTM BATCH SIZE
The proposed model's time complexity and memory consumption are affected by the batch size and the optimal LSTM batch size setting can be determined based on balancing the results between memory efficiency and running time. This can be done by considering the prediction loss as shown in Fig. 7. It can be seen that, when the batch size is increased, the prediction (Loss) is decreased. Furthermore, the running time is increased when the batch size is increased. In the experimental results, the best batch size=100.

B. COMPARISONS
As shown in Fig. 8, we compared the CNN-LSTM model with other models: CNN [35], LSTM [36], RNN [40], and CNN-RNN [41] in terms of performance evaluation, MAPE, and RMSE. The results show that our CNN-LSTM method has the smallest prediction error compared with other methods according to RMSE = 39.69 and MAPE = 27.56.
Furthermore, the F1-score was utilized to evaluate the predication accuracy of our method and then compared with other methods: CNN [35] and LSTM [36].
where values of TP, TN , FP and FN represent the number of true positives, true negatives, false positives, and false negatives, respectively. Precision = TP TP+FP and Recall = TP TP+FN . as shown in Fig. 9, and F1-score is approximately 0.9359 which is better than utilizing CNN or LSTM only. Tables 2 and 3   We included 7 courses C i.=1,2,3,...,7 selected from each student record in Blackboard. These significant features are utilized as KPIs for the students. This helped to predict the students' study behaviors using a deep learning model based on CNN-LSTM. These features were examined for various preparatory year courses studied during the first and second semester. The proposed method with selected features was examined and evaluated using precision as the criterion as reflected in Fig. 6.

As mentioned in
The experiment as shown in Fig. 10 observed that the proposed CNN-LSTM method achieved a precision score of 94.2% using 7 features together, whereas the proposed method achieved 90.94% precision using only 3 features [F1, F2, F4]. These features represent login, time of reading course, and number of downloads. It shows how many students are interested in the selected courses. The proposed VOLUME 10, 2022 method achieved 92.6% precision using 2 other features [F5, F6]. These features represent the assessment behaviors of student. The precision score of 89.0% was achieved using another 2 features [F3, F7]. These features represent the participation behavior of students.

VI. CONCLUSION
This study utilized collected data generated by students' interactions with an LMS (i.e. Blackboard). We measured the effectiveness of our deep learning CNN-LSTM model for predicting student performance using prediction accuracy and prediction error. 7 × 7 features were selected for each student as an input of CNN layers. Three factors were regarded in the deep learning model to affect prediction accuracy and prediction error: size of CNN convolution filter, neurons in LSTM, and LSTM batch size. The limitation of the CNN-LSTM model is high time consumption when increasing the size of CNN layers, filters, and LSTM batch size. Furthermore, it is worth noting that different feature selection methods may be utilized to reveal student performance. Furthermore the CNN-LSTM deep learning model with multiple layers can learn more effectively and provides greater computing power but takes longer time to train. Thus, future studies could utilize a light weight, shallow deep learning model that offers low training time with reasonable computing power.