Choosing YouTube Videos for Self-Directed Learning

YouTube provides a vital source for self-directed learning. YouTube’s search engine, however, ranks videos according to popularity, relevancy, and view history rather than quality. The effect of this ranking on learners’ behavior and experience is not clear: Do learners tend to choose from the top of the returned search list? Does the choosing behavior affect their learning? Is the type of sought knowledge relevant in this process? To answer these questions, we conducted two experiments with sophomore-level students in electrical and computer engineering programs. The students were asked to learn about two new topics by watching YouTube videos of their choice. The first topic conveys procedural knowledge about using the Quine McCluskey algorithm for minimizing logical functions. The second topic relates to the concept of the set-reset latch. In each learning session, the students had to report their watching behavior and experience by responding to an online questionnaire as well as to solve a problem related to the respective topic. The results show a clear tendency to choose from the top of the returned list. However, students’ performance in problem-solving was found to be uncorrelated with the choosing behavior. These results were similar for procedural and conceptual learning although the students’ performance in solving the conceptual problem was significantly lower. These findings indicate that university students who seek YouTube for self-directed learning can freely choose from the top of the returned search list without concern. There is no evident harm in doing so. However, students need to be thoughtful when using YouTube for conceptual learning. They should use different strategies such as watching multiple videos, selecting videos with higher viewer ratings, or watching videos with related procedural knowledge to support the learning of new concepts.


I. INTRODUCTION
YouTube is the world's second most-visited website [1]. With more than two billion users and one billion daily watching hours, YouTube is reaching millions of students worldwide [2], [3]. The first YouTube educational channel YouTube EDU was created in 2009 [4]. Since then, YouTube has been increasingly incorporated into formal education and classrooms [5]- [7]. This platform has gained increased attention in the time of COVID-19 to support teachers and students in distance learning [8]- [10]. Many studies have confirmed that the incorporation of YouTube in classrooms has promoted students' engagement, understanding, and overall satisfaction [11]- [13]. Moreover, it improved students' attitudes towards content and learning [11], [13] and increased students' overall course performance [14]- [16]. When using videos in typical classroom settings, students The associate editor coordinating the review of this manuscript and approving it for publication was James Harland. are provided with high-quality, class-related videos that are selected, evaluated, or created by instructors [17]- [19]. In the literature, several studies identified guidelines for instructors to search and identify appropriate YouTube videos for classrooms [20], [21].
Beyond classrooms and teachers' guidance, students also seek YouTube for self-directed or self-regulated learning [22], [23]. Moghavvemi et al. found out that students watch YouTube videos for entertainment, information seeking, and academic learning purposes [24]. Balakrishnan highlighted that students visit the YouTube site to enrich their learning activities and to communicate with others [25]. Rosenthal found out that the significant predictors of interest in YouTube are: (i) seeking related subjective norm, (ii) enjoyment of science, and (iii) informational use of YouTube [26]. In general, the role of YouTube and other innovative technologies in facilitating self-directed learning is a relevant question in the field of computers in humans' behavior [27], [28]. VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ When students seek YouTube for self-directed learning, they frequently make use of its search engine to discover the desired videos [22], [29]. They first type related search terms in the search field, hit the search button, and select videos from the returned list. However, considering the ever-increasing number of videos on YouTube, the students may face two issues: (i) formulating the accurate search query and (ii) choosing the right video from the returned list. Several studies have addressed the first issue in the literature [30], [31]. Burlington has asserted that not all learners have the sufficient skills to search YouTube [30].
Garett showed that learners tend to use simple, general rather than specific search terms to access YouTube videos and that this behavior leads to sub-optimal search results [23]. The authors recommended that content creators should be thoughtful while labeling their videos. For example, problem-based titles like ''Car Clicks Instead of Starting'' can be more helpful than technical terms such as ''Battery Corrosion.'' Also, the authors suggested that learners should be taught how to properly search for content. In a formal education setting, for example, students should be trained to use course terminology in their search queries [23].
So far, the second issue, which is choosing from the returned list, has not received sufficient attention in educational contexts. Yet, some researchers have identified that students follow different strategies to assess the quality of the returned videos. One strategy is to view the returned videos in sequence until they identify the one that addresses what they are looking for [20], [32]. Another strategy is to use a video recommended by their instructors as a reference for judging the quality of other videos. Furthermore, some students rely on the comments and videos' ratings as an indicator for content quality [20], [32].
Several researchers were trying to investigate and understand the factors that impact the popularity dynamics of YouTube and other social media platforms [33], [34]. Yet, there is a growing skeptical or critical position against the popularity-based ranking of today's search and recommendation systems [35], [36]. Chelaru found out that the top-10 videos in the list returned by the YouTube search engine, receive a higher number of views, likes, and comments [37]. The authors attributed this to the preferential attachment process (Yule process), which describes how individuals, who are already wealthy, receive more than those who have less. So, the YouTube algorithm is believed to promote videos that appear at the top at the cost of videos that appear below. This feature can be especially problematic for educational content since there is an increasing number of studies, which confirm a poor correlation between the popularity and the content quality of educational videos on YouTube [33], [34], [38], [39]. The authors of [40], found that among the first 10-most viewed videos, four only were of satisfactory quality. The exposure to unverified and partly misleading content [41], [42], makes it difficult for informal or unguided learners to navigate through YouTube [20]. Most of these studies focus on health-related information and medical content for informal learning [43]. Little is known about related issues in formal education. So, we don't know how students select from popularity-ranked videos for self-directed learning on YouTube, nor how the choosing behavior affects their learning. Answering these questions is relevant because it contributes to our understanding of self-directed learning as a core skill in general [44], [45] and the ability of YouTube to support this learning style in critical times [46]- [49].
The presented study aims to shed light on students' behavior while choosing videos from a search list and on the impact of this behavior on their learning performance in the context of self-directed learning. Specifically, the following research questions are addressed: 1) How do students select videos from a list returned by the YouTube search engine? 2) Does the selection behavior affect student learning? 3) Is the type of sought knowledge, procedural or conceptual, relevant in this context?

A. EXPERIMENTAL DESIGN
A total of 66 sophomore-level students (50% female, 50% male) were involved in this study. The participants were enrolled in a digital logic design course in Spring 2020. We conducted the experiment in the lockdown period, where students learned from home. Fig. 1 illustrates the research design as a flowchart where the main activities of the study are identified. In two separate sessions, we asked the students to learn about a new topic by watching YouTube videos and to solve a related problem that we posted on Moodle as an ungraded quiz. We provided the students with the search terms to make sure that they get similar video lists to select from. This allowed us to study students' selection behavior and analyze their perceptions of the same videos. We regard the learning in these sessions as self-directed because the students were given the freedom to select and watch as many videos as they want from the returned list. The students were mainly looking for answers to the provided problems. Thus, they kept watching videos until they were able to complete the quiz activity. The students were requested to keep a record of the watched videos and their ranking in the returned list. Also, they were asked to provide information on how long they watched each video and how useful the video was for solving the problem.

1) SELECTED TOPICS
Students frequently seek YouTube to learn how to solve a problem, i.e., to acquire procedural knowledge [50], [51]. Procedural learning is the acquisition of a procedure that consists of a series of steps, or actions, done to accomplish a goal [52]. Less is known about using YouTube for learning new concepts. Conceptual learning is defined as the ''acquisition and application of new knowledge to result in concepts and symbolic representations not previously in the individual's knowledge network'' [53]. In our study, therefore, we selected two topics that require procedural or conceptual learning, respectively. The Quine McCluskey method (QMS) is a multi-step algorithm for minimizing logical functions. So, the knowledge the students gain by learning this method is of procedural type essentially. Indeed, QMC relies on several concepts such as minterms, prime implicants, and minimization. However, our students learned about these concepts in previous sessions in the context of Boolean algebra and k-maps. The second topic is about the SR-latch, which is a simple logic circuit used to introduce the concept of one-bit memory. Understanding the SR-latch is a key for learning more advanced sequential logic elements and related concepts such as the state. Latches and flip-flops are known to be associated with several misconceptions in digital logic design [54], [55].
2) PROBLEM SOLVING ACTIVITIES (QMC AND SR-LATCH Quizzes) Fig. 2 shows the learning activities as posted on Moodle. The QMC quiz has six questions related to function minimization. After answering each question, the students were able check the answer before proceeding to the next question. The SR-Latch quiz is a single multiple-choice question with 12 choices. While Moodle performs automatic grading, the students were aware that the grades are not counted since the quizzes were meant to be learning activities.

3) QMC AND SR-LATCH VIDEO QUESTIONNAIRES
For each topic, the students were asked to provide information about the number of videos they, fully or partially, watched after finishing the quizzes. Also, every student had to name and rate up to three videos, which the student regarded as the best. Appendix A shows the questionnaire used to collect this information. For the videos' ratings, we used the criteria specified in [56], which include the content, the explanation quality, the technical presentation, the speaker's voice and language, and the video's length.

B. COLLECTED DATA
From the quiz attempts and the questionnaires, we collected the following data for every student: 1) Student performance in the quiz activity as a grade out of 10.
2) The number of videos watched fully or partially.
3) The titles of up to three videos the student rated best. 4) Student rating of these videos. At the time of the learning sessions, one of the coauthors accessed YouTube and searched for videos using the same search terms that the students were asked to use. From each returned list, the coauthor recorded the following data: 1) The titles and the ranks of the first 10 videos.
2) The number of views, likes, and dislikes, the upload year, and the video length.

C. PERFORMED ANALYSES
We used descriptive statistics to analyze the number of videos watched by the students and to aggregate their video ranking inputs. The relationship between the videos rankings in the returned lists and the number of views or the likes/dislikes ratio was analyzed using histograms. The same methodology was used to study the students' selections as a function of video rank. Correlation analyses were performed to understand the relationship between student performance and the ranks of the videos they watched. A t-test was conducted to analyze the effect of the type of knowledge on student performance.

D. A NOTE ON THE SAMPLE SIZE
Experimental research in education is frequently limited by the availability of a high number of subjects that can do the experiments under similar or identical conditions to support reliability [57]. In our case, 66 is around the number of students who usually enroll in digital logic design per semester. Repeating the test in multiple semesters would very likely affect the reliability of the research. In particular, YouTube is a highly dynamic platform. So, it should be expected that returned videos' lists will differ considerably from one semester to another. Another issue relates to the settings of our experiments and the fact that students have to solve new problems. Since the solutions to these problems are available to students after the experiments, it should be expected that they are also available to some students in the following semesters. So, new students would not do the experiments under similar conditions On the other hand, researchers believe that there is no golden rule for estimating the minimum sample size and that this largely depends on the research design [58]. One aspect  to be considered is the type of statistical analyses that need to be performed [57]. In our study, we performed a t-test and a correlation analysis. The t-test was used to compare student performance in conceptual and procedural learning in two experiments. So, we were interested in comparing two means. In this case, the sample size largely depends on the difference between the two means and the pooled standard deviation. When the mean values are distinctly different and the pooled standard deviation is small enough to discriminate the points, a small sample can be used without affecting the power of the statistical test. In our study, the grade means are 8.69 and 3.9 and the pooled standard deviation is 2.72. In this case, the sample size can be as small as 12 (6 in each experiment) to achieve a statistical power of 80% and a level of significance of 5% [59]. As for the correlation analysis, the sample size can be estimated based on the expected correlation coefficient and the probability of failing to reject the null hypothesis under the alternative hypothesis that is typically 0 or 0.2. Since we expect YouTube to rank higher-quality videos first, we choose a moderate value for the expected correlation coefficient, i.e., 0.5. In this case, a sample size of 29 would be sufficient [60].
In summary, while the sample size in our study is hard to increase, it is still sufficient for the performed statistical analyses. Table 1 and Table 2 summarize the features of the top-10 videos returned by YouTube according to the search queries described in the previous section for procedural and conceptual learning, respectively. These features include the video ID, the upload year, the number of views, the number of likes, the number of dislikes, the calculated Likes/Dislikes ratio, and the length of the videos. As can be seen from these tables and the diagrams in Figures 3 and 4, the top-ranked video on both lists has the highest number of views but belowaverage Likes/Dislikes ratio. The number of views tend to decrease with lower ranks as illustrated in Figures 3 and 4 for both cases. Despite looking arbitrary, the Likes/Dislikes ratio for videos with procedural knowledge tends to increase for lower-rank videos. In contrast, this ratio for videos with conceptual knowledge tends to be higher for higher-ranked videos. Table 3 and Table 4 show the number of watched videos per student for procedural and conceptual learning. In the first experiment, every student watched, on average, 1.8 videos to the end and 1.3 videos partly to learn about the procedure of minimizing logic functions using the Quine-McClusky algorithm. The median number of videos watched (completely or partly) is 3. As for conceptual learning, the students watched slightly fewer videos. On average, every student watched 1.5 videos fully and 1.2 videos partly to learn about the SR-latch. Table 5 and Table 6 show the average rating the students gave to the top-3 videos according to their ranking, for procedural and conceptual learning, respectively. It is important to highlight that Place-1 video, Place-2 video, and Place-3 video refer to any video that a student ranked first, second or third, respectively. So, the number 4.28 in the first column and first row of Table 5, for example, represents the average rating for the content coverage for all the videos which the students ranked first. Similarly, 3.81 in the second column and second row of Table 5 represents the average rating for the explanation quality for all the videos in which the students ranked second, and so on. In the last column, we calculated the average value of the ratings in the five categories. Interestingly, with only a few exceptions, the students' ratings are consistent in the sense that, in each category, Place-1 video obtained a higher rating than Place-2 video, which in turn has a higher rating than Place-3 video. Another interesting result is that the videos related to conceptual learning were rated higher, than the videos related to procedural learning, on average.  Table 1. Accordingly, the top two and top ten   videos in the returned list were chosen 59% or 89% of all the selections, respectively. The videos, which were listed at position 11 and below, were chosen just 11% of all the selections.

D. VIDEOS CHOOSING BEHAVIOR
As for conceptual learning, Fig. 6 shows that the top three videos in the returned list given in Table 2, were chosen 84% of all the selections. The videos, which were listed at position 11 and below, were chosen just 6% of all the selections.

E. STUDENT PERFORMANCE VS. CHOOSING BEHAVIOR
To understand the effect of the choosing behavior (which videos are chosen from the ranked list) on students' learning we performed a correlation analysis. For this purpose, we correlated students' grades in the respective problem-solving activity with the average rank of the videos they watched. We refer to the latter variable as the Choosing Tendency. To explain this variable, assume that a student has watched the videos ranked 1, 3, and 5. The choosing tendency is computed as (1 + 3 + 5)/3 = 3, which means that this student tends to choose the video at rank 3. If a student has viewed two videos ranked 2 and 6, the choosing tendency would be 4, in this case.
The correlation between the choosing tendency and the quiz grade is summarized in Table 7 for the procedural and conceptual learning activities. As can be seen from this table, the way students chose from the returned list is not correlated with their performance in the learning activity. In other words, there is no evident advantage or disadvantage for the selection from the top. Table 8 summarizes students' performance in the problemsolving activities for procedural and conceptual learning.      Accordingly, watching YouTube videos for procedural knowledge helped students perform significantly higher on the related activity (M = 8.69, SD = 1.94) compared to watching videos with conceptual knowledge (M = 3.91, SD = 3.31). The t-test confirms this effect t(47) = 2.0, (p < 10 −5 ) with Cohen's effect size d = 1.76. Note that 47 is the number of students who completed both activities. The remainder of the students have either completed one or none of the activities and were, therefore, not included in the t-test.

IV. DISCUSSION
In this section, we discuss the findings of the study in light of the research questions and other aspects. The results show that the students tended to select from the top of the list returned by the YouTube search engine. Specifically, around 65% or 85% of the students chose to watch the first three videos on the lists for procedural learning and conceptual learning, respectively. This is in line with the findings of some previous work that addressed viewers' behavior in general, i.e. not specific to education. Chelaru and Krishnappa showed that the top-10 videos receive the highest number of views [37], [61]. The motivation for students to select from the top of the list can be associated with their attitudes towards modern search engines in general. For example, Ofcom asserted that teenagers believe that search engines rank results essentially according to relevancy, usefulness, and trustfulness [62]. Another motivation for choosing from the top of the list could be the comfort of doing so, given the uncertainty about the benefits of alternative behaviors such as scrolling down to check other videos. Such behavior can be linked to what is known as choice overload or overchoice, which describes the increasing difficulty of decision-making when the number of choices increases [63]. This concept, which was originally investigated in marketing and consumer research, has also been applied in educational contexts. Iyengar and Lepper showed that students, who were given more topics to choose from, were less motivated to do a related extra-credit assignment [64]. Furthermore, the essays they wrote were of lower quality than the ones written by a control group of students who had fewer topics to choose from.

B. RQ2-DOES THE SELECTION BEHAVIOR AFFECT STUDENT LEARNING?
The results show that the choosing behavior has a low correlation with the grades that the students obtained in the respective problem-solving activity. In other words, our data provide no evidence that students, who tend to select from the top of the list, perform higher or lower than those who tend to watch lower-ranked videos. This result has one plausible explanation: All or most of the videos which appeared in the list and were accessed by the students are of a comparable VOLUME 10, 2022 educational quality which led to the low impact of the choosing tendency on learning. Some findings of related work on content analysis support this explanation indirectly. In particular, researchers, who analyzed medical content and health-related information on YouTube, confirm a low relationship between quality and popularity [40], i.e., the rank in the search list. This means that videos that are listed higher are not necessarily of better or lower quality [38], [39].
It may be argued that student background and the difficulty level of the problem-solving activity may have contributed to the low correlation between the choosing behavior and the performance. We believe that the possibility that such factors have affected our findings is low. First, the selected topics are specific to digital logic design, which is a core course in our programs. Students attending this course have usually no background in this subject, in general, and in the selected topics, in particular. Furthermore, the students' grades in activities showed clear variation. In particular, the coefficient of variation of the grades (SD/Mean) is 85% and 22% for conceptual and procedural learning, respectively.

C. RQ3-IS THE TYPE OF SOUGHT KNOWLEDGE, PROCEDURAL OR CONCEPTUAL, RELEVANT IN THIS CONTEXT?
The results show that our students performed better in the problem-solving activity related to procedural knowledge compared to the one related to conceptual knowledge. In general, YouTube is known to support procedural learning where viewers can learn how to do things step by step [50]. In an educational context, Aragon reported that exposing dental students to procedural videos enhanced their performance in fixed prosthodontics [65]. Also, Song and Kapur showed an improvement in procedural learning when Grade-7 students watched videos in a flipped classroom [66]. As for conceptual learning, however, the authors found out that watching videos was only helpful when the students had discussed the concepts and solved related problems in the classroom in advance [66]. Rittle-Johnson and Schneider confirm that instructional methods should support both types of learning iteratively because the relation between them is believed to be bidirectional [52]. While the videos in [65], [66] were selected by instructors, our students watched videos of their own choice after using the instructor-suggested search terms.
The low performance in the related problem-solving activity indicates that using YouTube was less suitable for the self-directed acquisition of conceptual knowledge; in contrast to procedural learning. This suggests that purposeful guidelines and recommendations should be developed to help students learn concepts from YouTube. The literature on conceptual learning and how it relates to procedural learning can be helpful in this context. For example, students could use explicit techniques to improve their conceptual understanding while searching or watching videos on YouTube. These include constructing concept maps during or after watching the videos [67], watching videos with related procedural knowledge, and trying to explain why the procedure works [68], as well as considering videos that promote the generation rather than the provision of new concepts [69].

D. OTHERS ASPECTS
The analysis of the returned search lists shows that videos are ranked according to the number of views essentially (Figures 3 and 4). The first video in the list has approximately as many views as the sum of the views of all other videos in the list for procedural knowledge, and even more for conceptual knowledge. This confirms the popularity-based ranking on YouTube as well as the previously mentioned preferential attachment process [37]. When it comes to viewer ratings, however, we can see that the most liked video in both cases is not in the top-3. In previous studies, we confirmed that viewer rating on YouTube is correlated with what students expect from educational videos [56], [70]. In other words, the Likes/Dislike ratio is an important quality metric of educational videos with college-level content. This suggests that students should pay more attention to highly liked videos for self-directed learning. We believe that more research is required to understand the effect of selecting videos according to viewer ratings on learning performance.
The evaluation of the videos by our students reveals two interesting aspects (Tables 5 and 6). First, regardless of how the students ranked their favorite videos, the ratings were highly consistent. So, their Place-1 video had a better rating than Place-2 video, which in turn had a better rating than Place-3 video in most rating categories (content, explanation, etc.) and for both procedural and conceptual knowledge videos. We regard this result as interesting because this level of rating consistency supports the reliability of our study to some extent. Such indication would be stronger if we could confirm that every student had rated the videos separately rather than comparatively. Unfortunately, in the experiments, we did not give the students any direction concerning this and we don't have any information about their rating behavior. More research would be desired to understand this aspect. Second, the videos with conceptual knowledge are rated higher than the videos with procedural knowledge, although student performance in the related activities was in the opposite direction. This indicates that the students' rating was not considerably affected by their performance in the learning activity. We are not aware whether the students rated the videos before, during, or after solving the problem. More research is needed to understand this aspect.

V. FINAL REMARKS AND STUDY LIMITATIONS
This study showed that students, who use the YouTube search engine for self-directed learning, tend to watch videos from the top of the returned list. This behavior showed neither positive nor negative impact on student performance. However, we found out that the learning performance is strongly affected by the type of sought knowledge: procedural or conceptual. Although students tended to rate videos with conceptual knowledge higher, their performance in the related learning activity was significantly lower than in the case of procedural knowledge. We also found out that the search list is ranked according to the number of views and not according to the Likes/Dislikes ratio. These findings can be used to make the following recommendations: 1) University students who seek YouTube for self-directly learning can freely select from the returned search list without big concern. There is no evident disadvantage of doing so. 2) Students need to be thoughtful when using YouTube for conceptual learning. They should use different strategies such as watching more videos, selecting videos with higher viewer ratings, or watching videos with related procedural knowledge to support the learning of new concepts.
Our study has some limitations which can be addressed in future research. The first limitation is related to the experiment setup, especially the number of students involved in the study. The second limitation is the focus on those videos which are on the top of the list, from which the students have selected essentially. It would be interesting to investigate the choosing behavior below this segment. Another limitation relates to the data collection. Recall that the students have named their best three videos. We relied on the fact that the median value of the number of watched videos is three to assume that the named videos are the ones selected from the list. Although demanding, the methodology could be improved by asking the students to provide the list of the videos they obtained from the YouTube search engine and to name all the videos they watched, not only the best three. VOLUME 10, 2022

VI. CONCLUSION
Students seek YouTube for entertainment, information seeking, social communication, and academic learning purposes. Also, YouTube is becoming a vital source for self-directed learning outside the classrooms. When students seek YouTube for self-directed learning, they frequently make use of its search engine to find videos with the desired content. While formulating the right search query has been addressed in the literature, student behavior while choosing videos from the returned list and the impact of this behavior on learning were still not clear. This study tried to answer these questions using an experimental research design. The results showed that students tend to choose from the top of the returned list. However, this behavior was uncorrelated with the learning performance regardless of the type of the sought knowledge, procedural or conceptual. Student performance in solving the conceptual problem was found to be significantly lower. These findings indicate that university students who seek YouTube for self-directly learning can freely choose from the top of the returned search list without concern. There is no evident harm in doing so. However, students need to be thoughtful when using YouTube for conceptual learning. They can use different strategies such as watching multiple videos, selecting videos with higher viewer ratings, or watching videos with related procedural knowledge to support the learning of new concepts. While this study provides first insights into student choosing behavior and its impact on learning, more research is needed to confirm these findings for a larger number of students and in other fields of study. Also, future research should investigate the viewing behavior itself, e.g., how long students stay on a video before they skip to another, how often they pause or rewind the video, and whether they read the video's descriptions or comments while watching. Screen recording and eye-gaze tracking can help address these questions.