Trajectories of engagement : a repeated cross-sectional investigation of student perceptions of an online learning environment

Evaluations of online learning environments (OLEs) often present a snapshot of system use. It has been identified in the literature that extended evaluation is required to reveal statistically significant developments in the evolution of system use over time. The research presented here draws on student OLE evaluations surveys run over the period 2004 2011 and include nearly 6800 responses exploring students’ perceptions of importance of, and satisfaction with elements of their OLE. Across the survey period, satisfaction ratings with all OLE elements rose significantly, suggesting a positive student engagement with the OLE over time. The corresponding ratings of importance of OLE elements generally rose significantly, though a number of elements registered no significant difference in the first two years of the survey, suggesting that short period surveys may struggle to reveal statistically significant trends. OLE element use appeared to be closely linked to perceived value. The OLE elements with the highest mean importance and satisfaction ratings related to student access of online learning resources. Other detailed results are also reported. We demonstrate a method for, and one large-scale case study of, quantifying and visualising the trajectories of engagement that students have had with an institutional OLE over time.


Introduction
In Australia, Deakin University is a major provider of distance and online education.In addition, it teaches on-campus at four campuses located in three cities in the State of Victoria.Initially, in the late 1970s, Deakin University saw itself as a major distance education provider, with some degree of separation between its teaching methods and materials used for on-campus teaching as opposed to offcampus teaching.The use of distance education methodologies and materials for both student cohorts gathered momentum in the early to mid-1990s under the strategic umbrella of flexible teaching and learning, and with a growing ''technological imperative'' (Holt and Thompson 1995) for the use of online systems for learning delivery and communication.In more recent times the university has implemented institution-wide online teaching and learning systems to provide opportunities to bring together all students in the one learning community.Iterating through a number of commercial learning management systems (LMSs), the university eventually settled on the WebCT LMS in 2003, branding it internally as Deakin Studies Online (DSO).The new LMS was trialled in 2003, and fully implemented in 2004.Concurrently, the university introduced policies requiring academic departments to migrate all online learning environment (OLE) activity to the centrally supported LMS Á at that time the LMS officially became the institutional OLE.Since that time, the OLE has expanded through the supplementing of the LMS with a range of satellite technologies including, synchronous communications, lecture recording and streaming, plagiarism detection, etc.; however, the LMS remains the core of the OLE.Another key initiative in the university's strategy to expand its online and distance education profile was to require that, from 2004, all its units of study have at least a basic online presence.Additionally, from 2004, all students enrolled in Deakin University undergraduate courses had to undertake at least one unit wholly online, with few exemptions given.In early 2006, WebCT was acquired by Blackboard Inc, leading to the phasing out of new development and support for the WebCT LMS.In 2010 Deakin University selected the Desire2Learn LMS as the replacement system for the WebCT/Blackboard LMS, and during 2011 commenced a phased cut-over to the new system, migrating content and users throughout the year so that by the first teaching period in 2012, all online support for teaching and learning would be provided via the OLE based on the new LMS.
OLEs are perhaps currently the most widely used and most expensive educational technology tool (Lonn and Teasley 2009;Salinas 2008;West, Waddoups, and Graham 2007), and like many other learning technology trends before them, have been adopted by institutions almost automatically, uncritically (Reynolds, Treharne, and Tripp 2003) and without evaluation of their effectiveness (Mahdizadeh, Biemans, and Mulder 2008).Where OLE evaluation is documented, it often presents a pointin-time snapshot of the use of the system, but does not capture the development of system use over time (Lonn and Teasley 2009).Extended evaluation of OLE usage is required to reveal the detail in the evolution of system use (Browne, Jenkins, and Walker 2006;Conrad 2005), and there is a call in the literature for the application of extended and/or repeated evaluation to better understand the impact of OLEs and to optimise their use in online teaching and learning (Bates and Khasawneh 2007;Davis and Wong 2007;Drennan, Kennedy, and Pisarski 2005;Mikropoulos and Natsis 2011).Existing published extended evaluations of OLEs are often those compiled by industry bodies and comprised of data drawn from across the sector (Browne, Jenkins, and Walker 2006;Smith and Caruso 2010).Such evaluations are valuable for sector benchmarking comparisons, but potentially subsume the nuances of the specific characteristics and context of individual institutions.
Given the scope of Deakin University's commitment (in terms of central infrastructure, policy development and roll-out of online components to all taught units) to online education, it was considered essential to evaluate the effectiveness of this investment.In 2003, a pilot survey of staff and students using DSO was conducted to establish perceptions of importance and satisfaction with various elements of the OLE.Following the full mainstreaming of DSO in 2004, the survey instrument was revised, and the survey process was expanded to include all Deakin University staff and students, and repeated again in 2005.The survey was administered using a university online survey tool.These surveys produced a large pool of data that provided insights into the initial engagement with the institutional OLE by students and staff, and some aspects of the student survey results have been reported previously (Challis 2005;Palmer and Holt 2010).
This previous research covered only the first two years of the full-scale system roll out.While evaluation of two years of operation can reveal interesting trends (Lonn and Teasley 2009), the observed differences may only be minor and/or statistically insignificant (Smith and Caruso 2010).A longer evaluation time frame may be required to reveal the deepest insights into the adoption and use of a new OLE (Davis and Wong 2007).Evaluation surveys that include consistent question items over a longer period of time permit more robust extended analysis of data trends (Browne, Jenkins and Walker 2006;Smith and Caruso 2010).In 2011, a new DSO evaluation survey was developed and administered to all students and staff.Crucially, the set of question items relating to use and perception of the core elements of the OLE was largely common with the previous surveys run in 2004 and 2005.This most recent evaluation survey serves two important purposes.Firstly as a measure of the current state-of-play with the current (but to be retired) LMS, it provides a benchmark measure against which the new LMS can be compared to at the point it eventually comes into full-scale use.Secondly, it provides a bookend measure of the development of student and staff perceptions of the value of the existing OLE across the seven-year period between 2004 and 2011.
The research presented here focuses on the latter aspect Á drawing on the large (nearly 6800 survey responses in total) and representative samples of student responses to the common DSO evaluation survey questions relating to perceptions of importance and satisfaction with core OLE elements in the years 2004, 2005 and 2011.This research responds to the call in the literature for additional extended investigation of OLE impact, and contributes to the literature by documenting a significant repeated cross-sectional institutional OLE evaluation Á both in terms of respondent numbers and time period covered.Statistically significant changes in student perceptions of the OLE over time are identified; the magnitude and direction of these changes is explored; and the nature of these changes are analysed.We provide a method for, and one large-scale case study of, quantifying and visualising the trajectories of engagement over time that students have had with an institutional OLE.

Methodology
Full details of the 2004 and 2005 DSO evaluation survey, its methodology, respondent samples and results have been presented previously (Palmer and Holt 2010).Broadly, all three instances of the DSO evaluation survey sought responses from students relating to: . demographic and background information; . perception of importance and satisfaction with a range of OLE elements; . a number of overall OLE satisfaction measures; and .open-ended written comments about the OLE.
The demographic information was used to test whether the sample respondent group was representative of the overall population of enrolled students.All three surveys

Research in Learning Technology
Citation: Research in Learning Technology 2012; 20: 17143 -http://dx.doi.org/10.3402/rlt.v20i0.17143contained 13 common OLE elements for which respondents were asked to indicate both their rating of importance and their level of satisfaction using ordinal response scales.Response scales of 1Á7 were used in 2004 and 2005, and a response scale of 1Á5 was used in 2011.The 13 common OLE elements were: 1. Accessing unit guide and other unit information; 2. Accessing unit lecture, tutorial or lab notes etc.; 3. Interacting with unit learning resources; 4. Using the unit calendar; 5. Contacting teachers via internal unit messaging; 6. Contacting students via internal unit messaging; 7. Reading contributions to online discussions; 8. Contributing to online discussions; 9. Completing online quizzes/tests; 10.Submitting assignments; 11.Receiving feedback on assignments; 12. Working collaboratively in a group; and 13.Reviewing unit progress.
All three surveys also asked respondents to indicate their level of agreement using a response scale of 1Á5 with a range of overall satisfaction measures relating to DSO.The overall satisfaction item ''the use of DSO enhanced my learning experience'' was included in all three surveys.The open-ended written comments collected in all three instances of the survey are a large and rich source of qualitative data in their own right, but are not included here due to practical space limitations.
This study did not attempt to follow a cohort of specific respondents over time Á for many students this would not be feasible.We sought to include students from all years of study, including those in the final year of their studies, and who would only be able to respond in one year.Even if initially surveyed in the first year of their studies, virtually no undergraduate students would still be enrolled in the same program of studies at the end of the period covered in this study.Instead, we sought a representative sample of the student population in each survey year.Here we compare the ratings of importance and satisfaction for each of the 13 OLE elements and identify any statistically significant differences across the three surveys.We also develop a method for visualising the trajectory of the importancesatisfaction data across the three surveys.Finally we compare the level of agreement with the common overall satisfaction item across the three surveys.Together these analyses provide insights into the development of student engagement with the OLE over an extended time period.As required by Deakin University human research ethics procedures, all of the surveys were anonymous and voluntary.

Findings and discussion
Response rate and demographic information Table 1 presents a summary of the response rates obtained in the 2004 and 2005 DSO evaluation surveys.The demographic match between the sample and population for both years was generally very good across the dimensions of gender, mode of study, level of study, enrolled faculty and enrolled campus.The full comparison has been detailed elsewhere (Palmer and Holt 2010).
In 2011, a new online system was used for the administration of the survey that saved all progressive responses entered, resulting in differential response rates for different sections of the survey.The 2011 effective response rate for those completing the entire survey was 5.8%, although higher response rates were obtained for some sections of the survey.A range of demographic information was available for the overall Deakin University student population, as well as collected as part of the survey, including gender, enrolled faculty, enrolled campus and duration of current enrolment.This permitted a comparison between the respondent sample and the overall student population on these demographic dimensions, as presented in Table 2.
In the 2011, the survey was administered in the first teaching period and it was not possible to include newly commencing students in the population sample, hence the size of the target population group was somewhat reduced compared to 2004 and 2005.However, we attempted to quantify the possible impact of this slightly different population group on student perceptions of online aspects of their study.Deakin University conducts an evaluation of teaching of every offer of the majority of units of study.This evaluation asks students to indicate their level of agreement (on a scale of 1Á5) with a range of statements, including item 9 Á ''The online teaching and resources in this unit enhanced my learning experience''.Commencing students are most likely to be found enrolled in first level units of study offered in the first teaching period of the year.Using data from the Deakin University student evaluation of teaching database for an entire annual period (mid-2009 to mid-2010), we compared the mean ratings for item 9 for first level units offered in the first teaching period to those from all other units of study.The means were identical to the fourth significant figure (first level 03.7657; other 03.7656) and, following confirmation of homogeneity of variance, an analysis of variance (ANOVA) test indicated no significant difference between the ratings from both classes of units (F 1427 03)10 (6 ; p0.998).
Although the response rates obtained in all years were comparatively low, they were not unexpected for an online voluntary survey (Cook, Heath, and Thompson 2000).The generally good match between the sample and population demographic characteristics in all years, and the confirmation that commencing students do not hold significantly different views from other students about the value of online aspects of their study, suggest that we can have some confidence in drawing more general inferences from the respondent data.

Extended importance-satisfaction analysis
In all three years, the DSO evaluation survey asked respondents to rate the importance of, and their satisfaction with, a range of elements of the OLE at Deakin University.In 2004 and 2005, a rating scheme of 10 low importance/satisfaction and 70 high importance/satisfaction was used.In 2011 a rating scheme of 1Á5 was employed.Research indicates that re-scaling of scale item data is possible (Dawes 2008), and in the following analysis we have re-scaled the 2004 and 2005 importance and satisfaction data to the range 1Á5.For both importance and satisfaction a ''not applicable'' option was also provided to permit students not using a particular element to avoid having to provide a contrived rating.Table 3 provides a summary of the mean responses for the importance and satisfaction ratings from all three years, with the percentages of ''not applicable'' responses (which are the same for importance and satisfaction rating pairs) shown in parenthesis.
A definitive indication of the significance of the differences between the mean ratings for an item over the three surveys is obtained from an ANOVA test using mean item rating as the dependent variable and survey year as the grouping variable.A requirement for the ANOVA test is that the variation of the mean rating be similar in all three survey years.Levene's test of homogeneity of variance failed for most survey items, and in these circumstances a robust ANOVA test using the Welch test statistic was performed instead.Significant differences in mean ratings between years were observed for most survey items.To establish which year pairs have significant differences in mean ratings, post-hoc pair-wise testing was performed.Where equal variance was assumed, Tukey's ''honestly significant difference'' post-hoc test was used; where equal variance was not assumed, Tamhane's T2 post-hoc test was used.Based on a significance level of pB0.01,Table 4 indicates the presence of significance differences in mean survey item ratings between survey year pairs Á ''No'' indicates no significant difference; '' '' indicates a positive significant difference; and ''¡'' indicates a negative significant difference.
Figures 1 and 2 present one way to visualise the extended importance and satisfaction ratings for the OLE elements over the three DSO evaluation surveys.Corresponding year pair mean values of importance and satisfaction are plotted as points on a two dimensional chart of importance (vertical axis) versus satisfaction (horizontal axis).The points for all three years for each OLE element are joined to form a two-segment vector chain that describes the trajectory of the mean ratings of importance and satisfaction for that OLE element over the period of the three surveys.The numbering of the OLE element trajectories in the chart below are the same as those used above.Note that compressed scales are used on both axes.For purposes of clarity, the OLE elements have been separated into two figures to avoid overlapping of trajectory lines.To permit absolute comparisons between figures, both figures have been plotted using the same axes ranges.
For example, consider the vector chair for element 8 Á ''Contributing to online discussions'' shown in Figure 1.The 2004 importance and satisfaction mean rating pair is denoted by the diamond at the left end of the vector.In 2005, denoted by the circle at the mid-point of the vector, there was essentially no increase in the mean importance rating, but a significant increase in the mean satisfaction rating, A second marked difference from the general positive trend was that only one item returned a significantly lower mean rating across the longer duration survey year pairs (2004Á2011 and 2005Á2011), and that was the importance rating for the element ''Reviewing unit progress''.While this result suggests that there is an opportunity to better use the OLE as part of engaging students in self-managing their studies, it is noted that the absolute mean importance rating for this item remained relatively high.An important overall observation is that many of the mean importance ratings showed no significant increase over the single year observation period of 2004Á2005, but did show significant increases in mean rating over the longer timeframes of 2004Á2011 and 2005Á2011.This confirms observations elsewhere (Smith and Caruso 2010) and supports the proposition that extended evaluation of OLE usage is required to reveal the statistically significant details in the evolution of system use (Browne, Jenkins, and Walker 2006;Conrad 2005).
Considering the proportions of respondents in Table 3 choosing the ''not applicable'' rating for particular elements shows some distinctive decreases (hence presumed increases in element usage) over the period under consideration.The use by students of online discussions (posting and reading contributions), online quizzes, and online assignment submission and return, have all apparently increased in both absolute and proportional terms.This observation is presumably linked to an increased incorporation of these OLE elements into assessable student learning activities by academic staff.
The element with the lowest 2011 mean importance and satisfaction ratings is ''Using the unit calendar''.This result may be due to the fact that the internal LMS

Research in Learning Technology
Citation: Research in Learning Technology 2012; 20: 17143 -http://dx.doi.org/10.3402/rlt.v20i0.17143calendar system was largely unused, as it duplicates the function of, and does not integrate with, existing calendar systems used by students.The elements with the highest 2011 mean importance and satisfaction ratings are ''Accessing unit guide and other unit information'' and ''Accessing unit lecture, tutorial or lab notes etc.''.These two elements could be considered ''basic'' OLE elements, and an institution should aspire to get a satisfactory rating from students for these.The provision of learning materials electronically may not necessarily be seen as high value adding educational LMS function, but the high importance and satisfaction ratings attributed to these elements by students suggests that they nevertheless value these OLE functions.The element with the highest 2011 mean importance rating and the lowest 2011 mean satisfaction rating is ''Receiving feedback on assignments''.Given the critical importance of timely formative/progressive feedback for delivering information about progress and clarifying expected and actual performance, so as to influence students to take a proactive role in their learning and for their development as selfregulated learners (Nicol and MacFarlane-Dick 2006;Yorke 2003), this result should be of concern, and act as a flag for action that could have a positive impact on the contribution of the OLE to student learning.

Contribution to learning
In 2004 and 2005, the DSO evaluation survey asked students to indicate their level agreement on a scale of 1Á5 with the statement ''The use of DSO enhanced my learning experience'', and essentially the same question was included in the 2011 survey.The mean response ratings for all three years (with 95% confidence intervals estimated) are presented in Figure 3.Note that a compressed vertical scale is used.
Acknowledging the slightly different respondent groups in the two earlier surveys, and the slight difference in wording of the response question in 2011, there is a significant increase across the period of the three surveys in the perception of students that DSO contributes positively to their learning.This increase mirrors, and in fact exceeds on a percentage basis, the overall mean trend in increasing satisfaction ratings observed for the individual OLE elements.

General discussion
Recent data indicates a significant proportion of LMS owners are considering changing their LMS platform (Instructional Technology Council 2011).This may be due to the age of the existing system, changes in the teaching and learning context of the institution or the emergence of new LMS vendors, including the option of open source systems (EDUCAUSE Learning Initiative 2011).The passing of time means that existing contracts with vendors may expire, and this coupled with recently observed consolidation amongst system vendors and new entrants into the LMS marketplace mean that, as has happened at Deakin University, there will be some changeover of LMSs.The research presented here spans the entire production lifecycle of the retiring LMS at Deakin University, and in documenting the student perceptions of the OLE at the end of long period of system development and refinement; it also establishes a valuable baseline benchmark against which to compare the performance of the new LMS.To this end, it is planned to repeat the DSO evaluation survey for at least 2012 and 2013, as the use of the new LMS, as the underpinning core of the new DSO OLE, becomes mainstream.
We acknowledge that during the period included in this study, the university LMS has gone through a series of upgrades that have altered the operation of some of the functions included in the surveys.However, these changes were largely points of finesse rather than fundamental changes to the operation of the LMS functions, and the same core set of LMS functionality was available throughout the entire survey period.The achievement of a high level of overall satisfaction in 2011 begs the question whether such a high level of reported student satisfaction can be sustained through the migration to the new LMS, or whether the inevitable change and consequent reorientation required by students will cause a step down in perceived contribution to learning and/or any re-tracing of the importance-satisfaction trajectories.It is noted that the perceptions of teaching staff may interact with those of students in determining the ultimate impact of an OLE on student learning outcomes (McGill and Klobas 2009).Although not addressed here, a complementary set of data has also been collected from teaching staff at Deakin University.Planned future research will investigate the interaction between student and staff perceptions of elements of the OLE.

Conclusions
The data from a large, repeated cross-sectional and quantitative survey of student perceptions of elements of an OLE were analysed to reveal the extended development of student engagement with the OLE.Drawing on nearly 6800 survey responses over the period 2004Á2011, statistically significant changes in student perceptions of the OLE over time were identified; the magnitude and direction of these changes was explored and the nature of these changes was analysed.Across the duration of survey period, satisfaction ratings with all OLE elements rose significantly, suggesting a strong and positive student engagement with the OLE over time.The corresponding ratings of importance of OLE elements generally also rose significantly over the duration of the survey period, though a number of elements registered no significant difference in the first two years of the survey, providing support for the proposition that short period repeated surveys may struggle to reveal statistically significant trends.
One counter result registering no significant increase in mean importance rating related to an OLE element with limited use Á element use appears to be closely linked to perceived value.A second marked difference was a significant decline in importance rating for the element ''Reviewing unit progress'', suggesting an opportunity to better use the OLE as part of engaging students in self-managing their studies.Single year ''point'' observations from the end of the survey period in 2011 were made.The OLE elements with the highest mean importance and satisfaction ratings relate to student access of online learning resources.The OLE element with the highest mean importance rating and the lowest mean satisfaction rating is ''Receiving feedback on assignments'' Á here suggesting another opportunity to better use the OLE as part of engaging students in self-managing their studies.
We have also demonstrated a method for, and one large-scale case study of, quantifying and visualising the trajectories of engagement over time that students have had with an institutional OLE.The survey period documented here matches the production lifecycle of the current LMS at Deakin University, which is currently being progressively replaced by a new system.A key question going forward is whether the relatively high ratings of satisfaction observed at the end of the life of the current LMS are maintained across the transition to the new LMS, or whether the inevitable disruptions will lead to any retracing of the observed trajectories of engagement.

Figure 3 .
Figure 3. Mean agreement with 'The use of DSO enhanced my learning experience' 2004Á2011.

Table 2 .
Response rate and demographic information for 2011 DSO evaluation survey.

Table 1 .
Response summary for 2004 and 2005 DSO evaluation surveys.

Table 3 .
Mean importance and satisfaction ratings for 2004Mean importance and satisfaction ratings for  , 2005Mean importance and satisfaction ratings for   and 2011.   .