Figure 1. Line graphs of total hits and final grade in a module at Level 4.
ORIGINAL RESEARCH ARTICLE
Raj Kapur Shaha* and Linda Anne Barkasb
aDepartment of the Built Environment, Faculty of Technology and Environment, Liverpool John Moores University, Liverpool, UK;
bFaculty of Business Law and Tourism, University of Sunderland, Sunderland, UK
(Received: 27 March 2018; final version received: 3 September 2018; Published 20 December 2018)
In higher education, e-learning technology such as Blackboard (Bb) is widely used and has become a popular tool worldwide. It helps reduce the communication gap between students and tutors, without time and location constraints. The study of student engagement and the impact on performance is a key issue in higher educational research, so identifying how students use e-learning technology can help contribute to how to design e-learning materials that further support student engagement. This quantitative research study examined two undergraduate engineering modules. Utilising the Statistical Package for the Social Sciences, the number of clicks students made on Bb was assessed against their classroom attendance, engagement with activities and their performance in the final grade in the module assessment. The outcomes contribute to the developing literature on students’ interaction with online learning, by providing an insight into the way students’ use of e-learning materials influences their performance in their studies.
Keywords: Attendance; Blackboard; communication; engagement; e-learning technology; higher education; Blackboard clicks; Blackboard hits; performance
*Corresponding author. Email: r.shah@ljmu.ac.uk
Research in Learning Technology 2018. © 2018 R.K. Shah and L.A. Barkas. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.
Citation: Research in Learning Technology 2018, 26: 2070 - http://dx.doi.org/10.25304/rlt.v26.2070
This article presents the findings of an initial study that was undertaken to examine how the use of Blackboard (Bb) could identify the extent to which students utilised online materials in their engineering courses. The study was undertaken to explore if regular links to the online resources were related to their attendance and supported their engagement with their studies. The findings contribute to the literature on how the development of Web-based technologies can enhance students’ performance in higher education.
The ease of access to online materials has also become more common because of advancement in information technology via e-devices such as desktops, laptops and smartphones. Although new e-learning resources are appearing all the time in education, one of the most prolific virtual learning environments (VLEs) is Bb. Bb is a system that allows users to access it via a unique username and password to log into their subject modules or programme. It started as ‘one of leading commercial learning management systems and then shifted to wide use as a course management system software package in educational institutions (Guo, Zhang, and Guo 2016; Zidan 2015, p. 230).
Electronic learning through the World Wide Web, or e-learning via the Internet, as it is now more commonly known, has become possible because of the advancements in communication, networking and broadcast technologies. The use of electronic materials is heavily researched from a number of different perspectives (Flavin and Quintero 2018). For example, research by Hewitt and Stubbs (2017) examined how learning technology could help address law students’ anxiety about their studies and improve their self-efficacy. A study by Young and Nichols (2017) examined how academics embedded digital learning approaches into the curriculum. Throughout this extensive research, the debates surrounding the use of the Internet and related advanced technologies have acquired a number of different terms, which are frequently used interchangeably in the literature. These terms include blended learning, distance education or distance learning; online environment learning; Web-based instruction and more recently VLEs (Young and Nicols 2017). VLEs such as Bb, Canvas and Web-Chat (WebCT) can be available 24 h per day, all year around.
Universities may have many national and international students studying in their programmes; therefore, within this heavily competitive marketisation and internationalisation of higher education, they have to ensure that they stay up to date with the latest e-learning technologies to improve communications, as well as student engagement and performance. This technology also helps to improve student engagement in terms of the time spent on a task, quality of effort and student involvement. The challenges and benefits of e-learning have been discussed in many articles (see inter alia Altuna and Lareki 2015; Bouhnik and Marcus 2006; Liaw, Huang, and Chen 2007; Raab, Ellis, and Abdon 2002), but a common thread throughout the research is the importance of e-learning technologies as a support mechanism for helping students to engage in their studies. Starting with a discussion of what is meant by engagement, the following section highlights some of the issues surrounding the use of e-learning technologies.
The generic term engagement employed throughout the literature on higher education depicts students’ study patterns, how they use their time, resources, relationships and communications with their tutors, peers and the organisation (Kahn 2014; Trowler 2010). Theories of how best to do this, however, vary across and within disciplines. From a behavioural perspective, engagement is defined as the ‘time and effort students devote to educationally purposeful activities’ (ACER 2010), but from a psychological perspective cognition incorporates individual characteristics such as motivation, self-efficacy and expectations as part of student engagement (Jimerson, Campos, and Greif 2003). Researchers in the UK have proposed a more holistic definition: ‘The conception of engagement encompasses the perceptions, expectations and experience of being a student and the construction of being a student’ (Bryson, Hardy, and Hand 2009). Whichever definition is postulated, research into improving students’ engagement in their studies embraces all the quality enhancement and quality assurance processes, ensuing in the improvement of the educational experience (UK Quality Code for Higher Education 2012).
Some studies have examined students’ feelings and emotions surrounding the process of engagement. According to Harper and Quaye (2009), student engagement is more than just involvement or participation. It requires a positive frame of mind, ‘mood’ and ‘sense making’ in addition to physically active involvement in different types of activities within the academic environment. Acting without sentiment, engagement is just like participation; feeling engaged without acting is known as dissociation. Fredricks, Blumenfeld and Paris (2004) classified student engagement into three dimensions (see Table 1).
A study by Stewart, Stott and Nuttall (2011) about the relationship between student engagement in terms of attendance, online learning and performance was inconclusive, but their findings did demonstrate the importance of attendance as a predictor of performance and argue it is influenced by the study behaviour rather than time spent on accessing the resources (Bb clicks, or ‘hits’), particularly online resources. They also suggested that an integrated blended learning approach could help to improve student performance.
While research into helping students engage with their studies has shown the importance of good communications, starting with clear guidance to students about what it is they will study, assessment and feedback (Higher Education Academy 2017; Kahu 2013; Thomas 2012), the complexity of this process is articulated in research findings across both the general and specialist literature on higher education (Zepke 2014). The emerging research into how students think and feel about their studies has also added to the intricacies of the debates whilst contributing to how different resources might be used in various ways to positively enhance the students’ experience and performance (Hewitt and Stubbs 2017). The stronger the engagement, the better the student is seen to perform (Trowler 2010).
The student profile in higher education has changed considerably over the past two decades, not only with the internationalisation of the curriculum (Higher Education Academy 2017) but with the attendance patterns of students. While the traditional, full-time student remains, many students now work part-time or combine distance learning with course attendance. This change in study patterns has necessitated the use of Web-based technologies.
The study concentrated on exploring the relationship that might exist between student engagement, attendance and performance. The study was limited to one undergraduate course module in a civil engineering programme, over two levels: Level 4 and Level 6. As part of their programme studies, students are normally required to search the learning and teaching materials for the coursework assignment and exam purposes. It is not possible, therefore, to argue that Bb hit rates have any impact on students’ engagement and progression with their learning, but it is possible to see how the hits linked with attendance and final performance, and this is useful to the module tutors to help them design the more effective online materials. While the insights from the study are limited to the exploration of the interaction with Bb on two engineering modules, and without further examination across other subject areas, no claim to generalisability of the findings can be made; nonetheless, the approach to the data collection and the findings may help to assist tutors and programme managers when designing module guidelines and structuring course materials.
A quantitative approach to data collection was employed. According to Aliaga and Gunderson (2000), the quantitative method is defined as ‘explaining phenomena by collecting and analysing the numerical data through mathematically based methods in particular statistics’. Quantitative methods are frequently described as deductive in nature, in the sense that inferences from tests of statistical hypotheses lead to general inferences about characteristics of a population (Bryman and Bell 2015).
In this study, it was hypothesised that student engagement via Bb hit rates has a significant relationship or correlation with class attendance, engagement and performance.
The study’s aim was to explore any connection between the students’ Bb hits and their attendance in their programme of study, engagement and performance. A statistical analysis test for the correlation between students’ online activities via Bb hits and class attendance was performed to understand the depth of the relationship between student engagement and its impact on student performance. This relationship will help inform further research into how best to enhance teaching and learning practices through the redesign of the module structure, inform guidelines and assist us in understanding the way students utilise online learning resources via the Bb system.
To minimise the impact of subject type and student cohort, two different course modules were included in the study with two levels of student performance in the civil engineering programmes. The study was based on secondary data analysis, which was gathered from the university Bb system and attendance records to reflect student use of online resources and physical participation in the classrooms.
The design of the study contained two aspects. The first aspect of the study aimed to examine the correlation between student engagement via online activities measured through Bb hit rates and student module performance. The online activities or hits were recorded under the course evaluation tool in the Bb system, based on the use of electronic resources, over the whole academic year in a course module titled ‘Construction Practice’ at Level 4 and the ‘Risk Management’ module at Level 6 in the civil engineering programme. The aim of the second aspect of the study was to identify any correlation that existed between class attendance and module performance of the student at Level 4 and Level 6.
The total number of students included in the study was 82 and 88 at Levels 4 and 6, respectively. The details of Bb clicks, records of class attendance and the final grade of each student on the module, at both levels, are shown in Appendix A. The secondary data for statistical analysis in the study was collected under three aspects or attributes of student engagement as detailed below:
The results of the data analysis are presented in the tables and graphs. Firstly, student engagement in terms of the Bb clicks and performance in relation to the final grade of students at Level 4 in the course module was analysed using SPSS. Two frequency graphs with student ID and mean values of Bb hits, attendance and final grade were drawn (Figures 1 and 2). Figure 1 reveals that there is a similar trend of fluctuation between student engagement and student performance but fails to identify what types of correlation exists between them. Similarly, Figure 2 shows that there is a slightly different frequency between attendance and performance, but the line graph does not identify any type of existing correlation between them. Hence, a t-test was then conducted to identify the positive or negative correlation between student performance and engagement at both levels. The results of the paired sample statistics, that is, the paired sample correlation and paired sample test, are presented in Tables 2, 3 and 4.
Figure 1. Line graphs of total hits and final grade in a module at Level 4.
Figure 2. Line graphs of total hits, final grade and attendance in a module at Level 4.
The t-test results of the paired sample correlation analysis revealed that there is significant positive correlation between Bb hits and the final grade (0.52, p = 0.00 <0.05) and between attendance and final grade (0.59, p = 0.00 <0.05) (see Table 3). However, when the paired sample test was conducted at a 95% confidence level, it was found that student engagement in terms of Bb hits had highly significant correlation with performance with a positive t-value (t = 9.99, p = 0.00 <0.05). In contrast, the paired sample test between students’ attendance and final grades reveals an insignificant result with negative t-values (t = −1.32, p = 0.19>0.05). The details of the paired test results are shown in Table 4. Moreover, the results confirm that student performance was positively correlated with student engagement in terms of Bb hits compared to class attendance, as an initial finding from the study. An additional regression analysis using SPSS was conducted to understand the importance and effect on student performance from student engagement aspects.
A regression analysis with automatic linear modelling was then conducted to analyse the linear effect on student performance (final grade) from the aspect of student engagement indicators (such as Bb hits and attendance). The results of the regression analysis are shown in Figure 3. The student performance on the module at Level 4 (mean = 55.74, SD = 20.57 and N = 77) shows the linear effect with respect to Bb hits and attendance (see Figure 3). The linear modelling results reveal that online activities related to exam preparation have more consequence compared to online activities associated with coursework. Figure 4 shows that the estimated mean has a significant effect on the final grade (student performance) from the engagement aspects of Bb hits and attendance. This supports the argument that there exists a positive linear relationship between student engagement and performance. The linear relation of student performance with respect to Bb hits indicates that it was more significant than class attendance for the Level 4 module of the programme.
Figure 3. Effect on the final grade from exam and coursework at Level 4.
Figure 4. Estimated means chart of student performance with aspects of engagement such as total hits and attendance at Level 4.
The results of the data analysis are presented in the tables and graphs. Firstly, the student engagement aspect (in terms of Bb clicks vs final grade) and student performance aspect (in terms of attendance vs final grade) at Level 6 was analysed using SPSS. Two frequency graphs with student ID and mean values of Bb hits, attendance and final grade were drawn and these are presented in Figures 5 and 6. Figure 5 reveals that there is a similar trend of fluctuation between student engagement aspects and their performance, but from the line graph it is not possible to identify the types of correlation that exists between them. Similarly, Figure 6 shows a slightly different frequency between attendance and performance and the line graph does not show the correlations between them. Therefore, a t-test was conducted to identify the correlation between student performance and engagement at both levels. The statistical analysis of the t-test with the paired sample correlation were performed and the results are presented in Tables 5, 6 and 7. The t-test results of the paired sample show that significant correlation exists between the student engagement aspect of the Bb hits and the final grade (0.24, p = 0.02<0.05) but an insignificant correlation exists between student attendance and the final grade (0.06, p = 0.00<0.61) (please see Table 6). On the other hand, when a paired sample t-test was conducted at 95% confidence level, it was found that student engagement and performance was highly significant with a positive t-value (t = 16.93, p = 0.00<0.05), whereas the paired test between student attendance and the final grade showed significant results but a negative t-value (t = −4.16, p = 0.00 <0.05) (please see Table 7).
Figure 5. Line graphs of total hits and final grade on a module at Level 6.
Figure 6. Line graphs of total hits, final grade and attendance for a module at Level 6.
Figure 7. Effect on the final grade from exam and coursework at Level 6.
These results confirm that student performance has some relationship with Bb hits compared to student attendance. A regression analysis was also conducted with automatic linear modelling using SPSS to understand the importance and consequence on student performance from the engagement aspects. The results are shown in Figures 7 and 8.
Figure 8. Estimated means chart of student performance and the engagement aspects of Blackboard hits and attendance at Level 6.
Moreover, the results and discussions about the regression analysis, which was conducted with linear modelling, was aimed at identifying the type of relationship between student performance and the engagement aspects.
The linear modelling results shown in Figure 5 reveal that online activities via Bb hits are related to exams and have the most important impact compared to online activities in respect of coursework assignments. The student performance on the coursework assignment at Level 6 (mean = 56.74, SD = 18.60 and N = 88) indicates the linear relationship between Bb hits and attendance (please see Figure 7). Figure 8 also reveals that the estimated mean has a significant linear relationship on the final grade from the viewpoints of engagement indicators (BB clicks and attendance). This demonstrates the existence of a linear relationship between student engagement and performance. The linear relation of student performance with respect to Bb hits has less impact than class attendance at Level 6.
From the statistical analysis of the research data, the findings were significant at both Levels 4 and 6. Firstly, it was recognised that student performance had a positive correlation with student engagement from the aspect of Bb hits at both Levels 4 and 6, but the types and the levels of correlation were different at both levels. One of the results showed that class attendance at Level 4 was significantly related to student performance but it was insignificant at Level 6.
Secondly, the other key finding was that student engagement from the aspect of Bb hits had a significant and positive connection in improving student performance at both levels, but student engagement as measured by attendance had an insignificant impact on performance. The study results also confirmed that student engagement had a linear effect on the student performance from the regression analysis. This exposed the issue that students need to be involved more in online activities in order to improve their performance in a course module. From these results, it could be argued that the results might be different in other subject areas because of the nature and complexity of different modules, where various levels of online activities take place. For example, lab-based or field-based modules need active participation compared to class-based modules; however, online activities can help to improve student understanding and performance.
A review of the literature illustrates the range and complexity of advancements in Web-based technologies and reveals the equally diverse ways that students utilise the e-resources available to them (Wang 2015). In this study, the findings showed that student performance had a positive and significant correlation with student engagement at both Levels 4 and 6 in the civil engineering programme; however, both types and level of correlation were found to be diverse at both levels. While class attendance was significantly related to student performance at Level 4, the relationship was shown to be insignificant at Level 6; however, from the regression analysis test, the results also confirmed that student engagement showed a linear relationship. This suggests that students’ involvement in online activities could help to improve their performance on a module. Of course, when various levels of online activities take place in the programmes of study, it can be argued that the results might be variable in other modules because of the nature and complexity of different subject areas.
Since Marton and Säljö (1976) first introduced the concept that students take different approaches to how they learn a subject, the extensive and rich literature on all aspects of the student learning experience has contributed to the knowledge of the intricacy of students’ relationship with their own learning. Across and within different subject domains, students employ a range of deep, surface and strategic approaches to their studies (see inter alia Bryson and Hand 2007; Fielding 2006; Gibbs 1992; Holmes 2015). Emerging research on the use of digital technologies now explores the intersection between the convergence of learning theories and digital technologies (Altuna and Larek 2015), and implementing blended learning frameworks could be one of the ways forward in research into the advantages and challenges of e-learning (Adekola, Dale, and Gardiner 2017). The advancements in technology-enhanced learning and teaching over the past decade adds another dimension to this complex relationship, so how best to utilise electronic material to encourage students’ engagement with their studies remains an ongoing area for further research.
ACER. (2010) Doing More for Learning: Enhancing Engagement and Outcomes, Australasian Student Engagement Report, Australian Council for Educational Research, Camberwell.
Adekola, J., Dale, V. H. M. & Gardiner, K. (2017) ‘Development of an institutional framework to guide transitions into enhanced blended learning in higher education’, Research in Learning Technology, vol. 25, pp. 1–16. doi:10.25304/rlt.v25.1973.
Aliaga, M. & Gunderson, B. (2000) Interactive Statistics, 1st edn, Prentice Hall, London.
Altuna, J. & Lareki, A. (2015) ‘Analysis of the use of digital technologies in schools that implement different learning theories’, Journal of Educational Computing Research, vol. 53, no. 2, pp. 205–227. doi:1177/0735633115597869.
Bouhnik, D. & Marcus, T. (2006) ‘Interaction in distance-learning courses’, Journal of the American Society Information Science and Technology, vol. 57, no. 3, pp. 299–305.
Bryman, A. & Bell, E. (2015) Business Research Methods, 4th edn, Oxford University Press, Oxford.
Bryson, C. & Hand, L. (2007) ‘The role of engagement in inspiring teaching and learning’, Innovations in Education and Teaching International, vol. 44, no. 4, pp. 349–362. doi:10.1080/14703290701602748.
Bryson, C., Hardy, C. & Hand, L. (2009) ‘An in-depth investigation of students’ engagement throughout their first year in university’, Paper presented at UK National Transition Conference, 22–24 May, London.
Fielding, M. (2006) ‘Leadership, radical student engagement and the necessity for person-centred education’, International Journal of Leadership in Education, vol. 9, no. 4, pp. 299–313. doi:10.1080/13603120600895411.
Flavin, M. & Quintero, V. (2018) ‘UK higher education institutions’ technology-enhanced learning strategies from the perspective of disruptive innovation’, Research in Learning Technology, vol. 26. doi:10.25304/rlt.v26.1987.
Fredricks, J.A., Blumenfeld, P.C. & Paris, A.H. (2004) ‘School engagement: potential of the concept, state of the evidence’, Review of Educational Research, vol. 74, no. 1, pp. 59–109.
Gibbs, G. (1992) ‘Improving the quality of student learning through course design’, in Learning to Effect, ed R. Barnett, SRHE and Open University Press, Buckingham, pp. 149–165.
Guo, S., Zhang, G. & Guo, Y. (2016) ‘Social network analysis of 50 years of international collaboration in the research of educational technology’, Journal of Educational Computing Research, vol. 53, no. 4, pp. 499–518. doi:10.1177/0735633115611114.
Harper, S.R. & Quaye, S.J. (2009) ‘Beyond sameness, with engagement and outcomes for all’, in Student Engagement in Higher Education, eds S. R. Harper & S. J. Quaye, Routledge, New York, pp. 1–15.
Hewitt, A. & Stubbs, S. (2017) ‘Supporting law students’ skills development online – a strategy to improve skills and reduce student stress?’, Research in Learning Technology, vol. 25, pp. 1–24. doi:10.25304/rlt.v25.1786.
Higher Education Academy. (2017) Internationalisation of the Curriculum-Toolkit: HEA, [online], Available at: https://www.heacademy.ac.uk/individuals/student-success/toolkits/internationalising
Holmes, N. (2015) ‘Student perceptions of their learning and engagement in response to the use of continuous e-assessment in an undergraduate module’, Assessment & Evaluation in Higher Education, vol. 40, no. 1, pp. 1–14. doi:10.1080/02602938.2014.881978.
Jimerson, S., Campos, E. & Greif, J. (2003) ‘Towards an understanding of definitions and measures of school engagement and related terms’, The California School Psychologist, vol. 8, pp. 7–27.
Kahn, P. E. (2014) ‘Theorising student engagement in higher education’, British Educational Research Journal, vol. 40, no. 6, pp. 1005–1018.
Kahu, E. R. (2013) ‘Framing student engagement in higher education’, Studies in Higher Education, vol. 38, no. 5, pp. 758–773.
Liaw, S. S., Huang, H. M. & Chen, G. D. (2007) ‘An activity-theoretical approach to investigate learners’ factors toward e-learning systems’, Computers in Human Behaviour, vol. 23, pp. 1906–1920.
Marton, F. & Säljö, R. (1976) ‘On qualitative differences in learning-outcome and process’, British Journal of Educational Psychology, vol. 46, pp. 4–11.
Raab, R. T., Ellis, W. W. & Abdon, B. R. (2002) ‘Multispectral partnerships in e-learning, a potential force for improved human capital development in the Asia Pacifica’, Internet and Higher Education, vol. 4, pp. 217–229.
Stewart, M., Stott, T. & Nuttall, A. (2011) ‘Student engagement patterns over the duration of level 1 and level 3 geography modules: influences on student attendance, performance and use of online resources’, Journal of Geography in Higher Education, vol. 35, no. 1, pp. 47–65.
Thomas, L. (2012) Building Student Engagement and Belonging in Higher Education at a Time of Change: Final Report from the What Works? Student Retention and Success Programme, Paul Hamlyn Foundation, Higher Education Funding Council for England, The Higher Education Academy and Action on Access, London.
Trowler, V. (2010) Student Engagement: Literature Review, Higher Education Academy, York.
UK Quality Code for the Higher Education. (2012) Chapter 5, Student Engagement, Higher Education Academy, York.
Wang, A. I. (2015) ‘The wear out effect of a game-based student response system’, Computers and Education, vol. 82, pp. 217–227.
Young, S. & Nicols, H. (2017) ‘A reflexive evaluation of technology-enhanced learning’, Research in Learning Technology, vol. 25, pp. 1–13. doi:10.25304/rlt.v25.1998.
Zepke, N. (2014) ‘Student engagement research in higher education: questioning an academic orthodoxy’, Teaching in Higher Education, vol. 19, no. 6, pp. 697–708.
Zidan, T. (2015) ‘Teaching social work in an online environment’, Journal of Human Behaviour in the Social Environment, vol. 25, no. 3, pp. 228–235. doi:10.1018/10911359.2014.1003733.