Students’ video viewing habits during a flipped classroom course in engineering mathematics

Research in Learning Technology 2020. © 2020 K.L. Nielsen. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.


Introduction
The flipped classroom has seen an increase in popularity in higher education in the last years. While a traditional lecture style often entails receiving instructional content in the classroom, the students work with the same content outside of class in a flipped classroom. The in-class time is then used for more student-centred activities such as group work and problem-solving. Several studies have shown positive results from using a flipped classroom method, such as increased student performance (e.g. Foldnes 2016; Nouri 2016), enhanced critical thinking and creativity (Al-Zahrani 2015), and perceived improved time management skills (Fisher et al. 2017). However, some studies have shown mixed results and challenges with implementing flipped classroom such as less satisfied students compared with students in traditional lectures (Yough et al. 2017). Some students find it difficult to adjust to the teaching method (Mason, Shuman, and Cook 2013) and keep up with the amount of work that is needed before the lectures (Strayer 2012). As a result, they might not come prepared to the lectures (Chen 2016), which is a critical part of succeeding as a student in a flipped classroom (Fisher et al. 2017). A flipped classroom usually entails students working with the course material by watching pre-recorded videos before the lectures, and it is, therefore, important to investigate how students use these videos to better understand how to make optimal learning videos as well as how to best implement a flipped classroom. Previous studies on how students use online videos have shown that students tend to favour watching the videos before assignments and tests (Elliot and Neal 2016;McGowan and Hanna 2015), as well as before the final exam (Brady, Wong and Newton 2013). However, these studies were conducted in courses with traditional lectures where the students had access to lecture videos as supplement material. It is also important to investigate viewing behaviours in a flipped classroom setting since students in a flipped classroom are more reliant on watching the videos in the absence of traditional lectures. While some studies on flipped classroom have shown similar results with an increase in views before tests, assignments or exams (Ahn and Bir 2018;Boevé et al. 2017), there has also been a study that found different behaviours between students in a flipped classroom and traditional lectures (Walsh, Brien, and Slattery 2019). Walsh, Brien and Slattery (2019) investigated how business students in a flipped classroom and traditional lectures used the same online videos differently. They found that students in the traditional lectures used the online videos more actively than students in the flipped classroom, which is the opposite of what one might expect since the flipped classroom students had no lectures prior to in-class activities (Walsh, Brien, and Slattery 2019).
A common result in previous studies on student viewing behaviour is that students tend to watch less of longer videos (e.g. Guo, Kim, and Rubin 2014;Lin et al. 2017;Ozan and Ozarslan 2016). Guo, Kim and Rubin (2014) studied student watching behaviour based on 6.9 million video views in a Massive Open Online Course (MOOC). They found that student engagement with a video, which in their case meant how much of a video the students watched in a single viewing session, started decreasing dramatically when videos exceeded a length of 6-9 min. Based on their findings, they recommended videos to be 6 min or less to maximise student engagement (Guo, Kim, and Rubin 2014). However, as Ahn and Bir (2018) argued, campus university students might be different from MOOC students and it is therefore also important to study video usage from students that might be more representative for flipped classroom teachings in a university setting (Ahn and Bir 2018). Lagerstrom, Johanes and Ponsukcharoen (2015) argued against using the 6-min recommendation as a fixed rule for the same reason. Based on their findings, they recommended learning videos used in universities to have a video length of 12-20 min. However, they specified that these recommendations were meant as a 'rule-of-thumb' and not as a fixed rule. The 6-min rule has also been criticised by Geri, Winer and Zaks (2017) who showed that interactivity can increase student engagement with videos (Geri, Winer, and Zaks 2017).
While studies have shown that students watch less of longer videos, there are several ways to interpret the results. If the data show that the students watched on average 50% of a video in a viewing session does this mean that they never watched the last 50% of the video? It might seem that some researchers interpret the results this way. For instance, Brame (2016) cited the research by Guo, Kim and Rubin (2014), where maximum engagement was found for videos in the 6-to 9-min range and commented that 'making videos longer than 6-9 min is therefore likely to be wasted effort' (Brame 2016). This article aims to increase knowledge of student viewing habits in a flipped classroom in order to better understand how to effectively implement a flipped 3 (page number not for citation purpose) classroom. In addition, the article aims to shed light on the interpretation-problem of viewing percentage explained above as well as investigate the 6-min recommendation by Guo, Kim and Rubin (2014).

Background
The flipped classroom was utilised in a second semester engineering mathematical course at Sør-Trøndelag University College in Norway (now a part of the Norwegian University of Science and Technology). The class consisted of 118 students and was taught by the author of this article. A typical week for the students consisted of traditional 2 × 45 min theory lectures on Mondays, Tuesdays and Thursdays, in addition to sessions where students could receive help from experienced students with mandatory written assignments. To be eligible for the exam, a certain number of the written assignments needed to be approved.
The flipped classroom was utilised in the second half of the course and 104 learning videos were made to cover the curriculum. The videos followed the Kahn Academy style consisting of handwritten notes using a tablet and drawing software and varied in length from 3 to 35 min with an average of 13 min (standard deviation [SD] = 6.1 min). The videos were grouped into 'lessons' which usually contained 4-6 videos. The list below shows an example of a lesson.
(1) Taylor and Maclaurin series (2) Example 1 (3) Example 2 (4) Example 3 (5) Using Maclaurin series to simplify root expressions There were a total of 20 lessons, usually 3 per week to mirror the previous format of three lectures per week. 6. The lesson bundles had an average video running time of about 3 h. The videos could in broad strokes be categorised into videos which introduced a new concept, 'theory videos' and videos that focused on solving problems, 'example videos'. This distinction was granular, that is, some theory videos could have examples and example videos could, quite often, present new theorems that were relevant to solving the problem. Nevertheless, videos that had a focus on presenting new theory usually had a more descriptive name, such as the first video in the lesson described above, whereas example videos usually had a more generic name 'example 1, example 2' and so on. All videos were made available to the students on the learning management system 'itslearning'. When presenting an example video, the problems that were to be solved in the video were written in text form above the video. This was to allow students to try to solve the problems themselves before watching the video.
During the flipped classroom weeks, the traditional lectures were replaced with student group activities and individual problem-solving exercises and attending these substituted the previously written assignments. The video lessons were structured so that the videos that the students were supposed to watch during a specific week, which was specified on itslearning, contained theory that was relevant for the assignments for the following week (see Figure 1). Most of the videos were accessible from the first week of the flipped classroom in order not to restrict students who felt comfortable with the current content and wanted to start preparing for later content.
Half of the class attended group assignments on Mondays whereas the rest of the class were given 'study time' which they were encouraged to spend working with the videos for the next week's assignments to reduce the amount of work needed to be done out of class. On Tuesdays, the roles switched so that students who had group work on Mondays now had study time and vice versa. On Thursdays, all students participated in individual problem-solving exercises. The flipped classroom approach was used in the last 12 weeks of the course. However, during the first week (Day 1-7 in Figure 1) the students still attended the traditional lectures. The students were given a lecture during this first week which contained information about the flipped classroom and how to study effectively as a flipped classroom student. On Days 50 through 57, the students had Easter holiday and there were consequently no group assignments in Week 9 since the last day of the holiday ended on the Monday in Week 9 (Day 57). In addition, the instructor was away during the first part of Week 10 (Day 63-70), and as a result, there were only problem-solving exercises and no group work during Weeks 9 and 10. However, the students were encouraged to attend a voluntary group assignment (without assistance from the lecturer) in Week 10 (Days 64 and 65), which would count towards the number of assignments needed to be eligible for the exam. The exam was on Day 85.

Video statistics and surveys
Although the students used itslearning to access the videos, the videos themselves were uploaded on YouTube. There were three main statistics gathered from YouTube; number of views for different videos, how much of the videos were viewed per viewing sessions and audience retention. Audience retention shows number of views for every moment of a video as a percentage of the total number of video views (YouTube 2019). For example, if a video was watched two times where both watched the first half of the video but only one watched the second half, the first half of the video would have retention of 100%, whereas the second half would have retention of 50%. By rewinding a video and watching the same part of a video multiple times during the same viewing session, one could get audience retention rates of over 100%.
To minimise views from other sources than the students for reliable data for this research, the videos were uploaded as 'unlisted'. Unlisted videos are not publicly available unless you have a direct link to the videos. Although this minimises views from other sources than the students in this course, there is still the possibility that students shared the link with other students not a part of the flipped classroom study. The students were encouraged not to share the link with others until after the exam. In addition to video statistics, the students were asked to answer surveys on their experiences with studying as flipped classroom students. However, students' experiences with the flipped classroom will be explored in a different paper.

Results
The 104 videos had a total of 14 171 views and a watch time of 101 119 min in the 85 days of the flipped classroom. During the first five assignment weeks, there were large spike in views on Sundays and Mondays before group assignments (see Figure 2). After the Easter holiday, there was a spike in views on the Wednesday before the problem-solving exercise followed by a similar pattern as before with spike in views on Sundays before voluntary group assignment in Week 10 (Day 64) and ordinary group assignment in Week 11 (Day 70). As the flipped classroom progressed, older videos received only sporadic views, which increased slightly in intensity during the last week before the exam.
While half of the students were given Mondays and Tuesdays on the group assignment days to work with the video content for the sessions in the following week, Figure 2 indicates that most of the views on these days were on videos relevant to the current assignments. This is also supported by Figure 3, which shows the average view per video for the first five assignment weeks relative to the days where the students  were supposed to watch the videos. In other words, many students waited until right before the group assignments to watch the videos. Adding the average view count for all videos from Monday where students were supposed to start watching the videos through Sunday on the week after results in 122 views. While this is comparable to the number of students (118), the view count also includes possible multiple viewing instances from identical students. Figure 3 also shows the difference in average views on theory and example videos during the first five-session weeks. While the view count is similar for most days for theory and example videos, there is a large difference on Mondays on the first group assignment day with theory videos receiving a significantly larger amount of views (Cohen's d = 0.99, p < 0.001 with Student's t-test). During the flipped classroom as a whole, theory videos had an average of about 149 views while example videos had a significantly lower number with an average of about 123 views [p < 0.001 (Student's t-test) and effect size of 0.92 (Cohen's d)], see Figure 4. Although there were no questions on the survey that asked specifically about the difference between theory and example videos, one student expressed on the survey that it was 'easy to skip examples and less "important" things when you sit at home alone'.
Figure 4 also shows average views for different video position within a lesson. There was a negative correlation between average views and lecture position (Spearman's rho = -0.41). The first video, which was usually a theory video, had the most views with number of views dropping on average for videos later in each lesson. Removing the first video from the calculation of difference in views between theory and example videos, there is still a significant difference (p < 0.01) although the average views on theory videos now are slightly lower (141) while the average views on example videos remain the same. Even so, effect size is still quite large (Cohen's d = 0.65). Figure 4 shows that a few isolated videos that had lower views than the rest. Four videos were marked as 'for specially interested' and were not required to watch in order to complete the work assignments. These had the lowest view count of any videos. If these videos are excluded from the calculations between theory and example videos, assuming that many students did not see these videos as an important part of their curriculum, there is an even larger effect size in the difference in views (Cohen's d = 1.26) with an average view count of 153 and 125 for theory and example videos, respectively. Excluding videos placed first in a lesson, the effect size is reduced to 0.92 with an average view count of 149 and 123.  Figure 3. Average video statistics for Weeks 2 through 6 per video relative to the days where the students were supposed to watch the specific video. The vertical dotted lines represent days with mandatory assignments (group assignments on Monday and Tuesday, and problem-solving assignment on Thursday).

Percentage viewed and retention
The videos had an average view percentage, that is, how much of a video was watched in a single viewing session, of 57% (SD = 14%), with an average view time of 7.13 min. There was a strong negative correlation (Pearson's r = -0.81) between the length of a video and average view percentage, see Figure 5. The dotted line on the left figure in Figure 5 shows a least-square fitted regression line which shows that for each minute of video length the view percentage dropped by 1.8% on average. The right figure in Figure 5 shows a similar statistic but with average view duration instead of percentage viewed. Videos between 20 and 25 min had the highest average view duration with a median of 9.6 min, followed closely by videos between 15 and 20 min with a median of 9.4 min. There was no statistical difference on average view percentage on theory videos compared with example videos, nor were there any correlation between average view percentage and video lesson position. Figure 6 shows average audience retention for different video lengths as well as an example of a retention graph for a single video. At the beginning of each video, the audience retention is on average higher than 100% before falling sharply. After this sharp fall, the retention lines have relatively small slopes until the last part of the video where there is a higher falloff at the very end. Excluding the first and last 5% of   the videos, linear regression yielded slopes ranging from -0.054 (for videos of length over 20 min) to -0.103 (for videos between 15 and 20 min), that is, a drop-off in retention ranging from 4.9% to 9.3% for the main body of the videos (between the first and last 5%). For all videos independent of length, the average drop-off was 7.7% for the main body with a slope of -0.086. The retention graph for the single video shows spikes in retention at specific times. Although a rigorous analysis of such spikes was not performed, anecdotal findings might suggest that these occur at visually strategic points in a video, such as displaying theorems.

Discussion
The students in this study were encouraged to spread video watching throughout the week, but the results suggest that many students ended up watching most videos on the last day right before each assignment sessions. This is aligned with the study by Ahn and Bir (2018), which saw an increase in views before tests and assignments. However, the students in this study did not use the videos before the exam to the same extent as the study by Ahn and Bir (2018) and Boevé et al. (2017), which had the highest number of views right before the exam. That students do not use the videos as encouraged by the instructor can also be seen in the study by Kinsella, Mahon and Lillis (2017) where the students were supposed to watch videos before lectures, but rarely did.
Watching all videos on a single day induces a very heavy workload on the students and it is possible that many students in this study, who waited until the last day to watch the videos, opted to skip 'less important videos' to reduce the workload. The results in Figure 3 suggest that example videos were deemed as less important by these students, and hence were more likely to be skipped. This is aligned with the study by Seaton et al. (2014) where the students spent more time on lecture videos (theory videos) rather than on tutorials (example videos), as well as the study by Kim et al. (2014), which reported a higher dropout rate for tutorial videos compared to lecture videos. In a study by Lin et al. (2017), the students seemed to engage more with videos that contained concrete information relevant for solving specific tasks, such as solving problems (Lin et al. 2017). The result of Lin et al. (2017) raises the question why example videos could be perceived as less valuable by the students, as these were more likely to focus on how to solve a different kind of problems rather than presenting theoretical concepts. A possible explanation for the difference in views and perceived value of theory videos versus example videos in this study could be to the naming scheme used for these videos. While a theory video could be named 'Partial derivation of higher degrees', the following example videos would be named 'Example 1', 'Example 2' and so on. Even though the distinction between theory videos and example videos was granular (i.e. example videos could introduce new theorems and theory videos could show examples), example videos may have been perceived as less important because of their less descriptive names. The placement of a video within a 'lesson' could also be a factor that could have determined the value of a video as videos at the end of a lesson tended to have lesser views than videos at the front. It is possible that some students might have thought that these videos are less important since they are placed at the back of a lesson. A third possibility is that example videos had the problems that were to be solved in the video displayed in text above the video on itslearning. This might have indicated to some students that this video just solves these problems without introducing something new or 'important'.
While it can be easy to blame the students for poor study habits, it might be a result of not focusing enough on self-regulation and self-evaluation. Besides the initial lecture on how to study effectively as a flipped classroom student, there were no other assignments related to self-evaluation or metacognition. The students in this study were on their second semester, and thus most likely had not had the time to develop the necessary self-regulation techniques needed to adjust to the flipped classroom lecture format (Yilmaz and Baydas 2017). Students with poor study skills have also a higher tendency to fail to regulate their own learning (Lust, Elen, and Clarebout 2013). It is, therefore, important to not underestimate the importance of training students to develop skills in self-regulated learning and metacognition (Zimmerman 2002), for instance, by utilising self-evaluation surveys, if a flipped classroom approach is to be used. To minimise students skipping videos perceived as less valuable, Lin et al. (2017) argued that it is important for instructors to try to improve students' perceived value of watching the videos, for instance, by explaining the importance of the content in the videos and show a connection between the content and the student activities (Lin et al. 2017).

View percentage and audience retention
The results in this study show that students tended to watch less of longer videos per viewing sessions, which is aligned with previous studies (e.g. Guo, Kim, and Rubin 2014;Lin et al. 2017;McGowan and Hanna 2015;Ozan and Ozarslan 2016). However, the results shown in Figure 6 contradicts the notion that a view percentage of, for instance, 50% indicates that the students only watch half of the content in a video. Even though there is a slight downwards slope in viewing retention from a start of a video to the end, the shallow slopes indicate that the students, to a large degree, do watch the videos in their entirety, but not in a single view session. If the students did not watch later parts of the videos on average, which could be one way to interpret the average view percentage of 57% in this study, there should have been a steep downwards slope in view retention since the first parts of the videos would receive a much greater number of views compared with the later parts of a video. Instead, the results indicate that the students watch the videos in their entirety over multiple viewing sessions. The interpretation of a low view percentage meaning that the students do not watch later parts of a video could stem from this being more common in online videos in general. For instance, Wistia, a video hosting service, analysed videos hosted on their site and found that the middle parts of the videos (which they defined as the parts between the first and last 2%) had an average retention loss of about 40% for 5-to 10-min videos and closer to 50% for videos between 10 and 20 min (Currier and Fisherman 2015). In a flipped classroom, however, students have a greater incentive to watch the videos in their entirety since they form the curriculum of a course they are attending.
The 6-min recommendation proposed by Guo, Kim and Rubin (2014) was based on similar data as those shown in Figure 5. They observed that videos between 6 and 9 min had the largest average view duration independent of video length, in addition to noticing a sharp drop off in average view percentage for videos larger than 6-9 min. In this study, the largest view duration was found for videos between 20 and 25 min, although videos between 15 and 20 min followed closely. The results thus support Lagerstrom, Johanes and Ponsukcharoen (2015) in arguing against using the 6-min rule as a universal rule for all learning videos. The appropriate length of a video will most likely depend on the subject matter and student group. However, keeping a video short is still recommended since it is probably more beneficial to the students if they are able to watch the video in its entirety in a single view session.
Sudden spikes in retention, such as those present in Figure 6 (left), have been investigated by Kim et al. (2014). Their study showed that 61% of peaks were associated with a visual transition in the video, such as changing slides. Although there was no rigorous analysis of retention spikes in this article, the anecdotal findings of the research presented in this article are consistent with these results. When a student moves the mouse to the timeline in the YouTube video player, the student can see a small miniature screenshot of the video at the specified time. Visually clear transitions in the video are easy to spot when 'scrubbing' the timeline and therefore likely to be chosen as a starting point when continuing to watch a video that was started in a previous viewing session, or when rewatching important parts of a video. In the study by Kim et al. (2014), high spikes in retention were to a larger degree caused by students rewatching the videos rather than first-time viewers 'rewinding' the videos to rewatch sections of a video. While first-time viewers tended to view the videos more sequentially, students rewatching a video had a higher tendency to be more selective in which parts of the videos they watched . Finding specific parts in a video when rewatching the video would likely require more scrubbing than rewinding a minute or two to rewatch important sections during the same viewing session. To reduce the amount of scrubbing required by the students, it could be beneficial to provide students with time codes for points-of-interest in the videos (transition to new content, showing a theorem, etc.), especially if the content is presented in longer videos where students are likely to use multiple viewing sessions to watch a video from start to finish, or if the videos contain multiple points-of-interest that students are likely to watch multiple times.

Conclusion
This study investigated students' viewing behaviour in a flipped course in engineering mathematics based on statistics from YouTube Analytics. The results showed that 11 (page number not for citation purpose) the students tended to watch less of longer videos, but the results indicate that they to a large degree watch every part of the videos, just not in a single viewing session. Adding time codes to strategic points in the videos was recommended to make navigation easier when returning to a video. The longest average view duration was found for videos between 20 and 25 min in length, followed by videos between 15 and 20 min. Many students seemed to base their viewing decisions on the perceived value of each video, and the naming scheme of the videos was hypothesised to have influenced their perceived value. However, more research is needed on how the title of a video in flipped classroom influence students whether they watch the video or not. The statistics also indicated poor study habits of several students as many would wait until the day before assignments before watching the required videos, which results in a very high workload on the students. Since a flipped classroom requires many adjustments to study habits from traditional lectures, the author recommends focusing on learning students study techniques related to flipped classroom as well as using self-evaluation surveys or other tools related to developing self-regulation skills.