ORIGINAL RESEARCH ARTICLE

A Disruptive Innovation perspective on students’ opinions of online assessment

Michael Flavin*

Principal’s Office, King’s College London, London, UK

(Received: 12 January 2021; Accepted: 19 April 2021; Published: 28 May 2021)

This article analyses students’ thoughts and feelings about online assessment. This article uses Disruptive Innovation theory as a lens through which to analyse students’ responses to online assessment, in a case study of a Leadership course. The sources of data for this article comprise annual course evaluation surveys, a one-off assessment survey and a focus group. Qualitative content analysis with a directed approach is used to analyse the data. The results show students are capable of undertaking a range of online assessments but are, in general, reluctant to utilise the innovative possibilities of different forms of online assessment. This article adds to our understanding of online assessment by placing it within a distinct theoretical framework, offering explanations for why students may not be seeking-out innovative forms of assessment.

Keywords: online assessment; disruptive innovation; leadership; e-portfolios; case study

*Corresponding author. Email: michael.flavin@kcl.ac.uk

Research in Learning Technology 2021. © 2021 Michael Flavin. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.

Citation: Research in Learning Technology 2021, 29: 2611 - http://dx.doi.org/10.25304/rlt.v29.2611

Introduction

The Principal’s Global Leadership Award (PGLA) was introduced to King’s College London in 2015–16 as a second-year undergraduate course. A postgraduate version was introduced in 2017–18. The PGLA teaches leadership as a collaborative social practice rather than an individual cognitive trait. Each year it receives c.450 applications for 40 places. It is interdisciplinary: students from any of King’s nine faculties can apply. Five classes facilitated by established figures in leadership, including former government ministers, are underpinned by a virtual learning environment (VLE) and set readings, in a blended learning format. Assessment is undertaken online in a series of incremental tasks, comprising an e-portfolio.

This article analyses students’ opinions of online assessment on the PGLA. The specific questions posed are as follows:

This article begins by examining Disruptive Innovation theory, its core arguments, its development and how it has been critiqued. It then summarises the modes of assessment on the PGLA and the different levels of data gathering used in this study. This article then undertakes data analysis and presents its results. The discussion returns to Disruptive Innovation to evaluate online assessment on the course, showing how students are capable of meeting the technological challenges of online assessment but are less interested in undertaking innovative online assessment. This article also proposes reasons why students are not seeking out innovative forms of assessment, using Disruptive Innovation theory to frame the analysis.

Literature review

Disruptive Innovation (Christensen 1997; Christensen & Raynor 2003) is a theory about goods and services, originating in the Harvard Business School in the 1990s, especially in the work of Clayton Christensen (1952–2020). The theory is influential and has been applied to a range of practices beyond goods and services, including healthcare (Christensen et al. 2009) and higher education (Christensen & Eyring 2011). It argues simple, convenient and easy to use technologies (what Christensen [1997] calls ‘disruptive technologies’ [p. xv]) can displace more sophisticated incumbent technologies. Christensen positions disruptive technologies against sustaining technologies; the latter offering incremental enhancement along an established performance trajectory.

In a subsequent, co-authored study (Christensen & Raynor 2003), the term ‘disruptive technology’ was replaced by a broader theory ‘Disruptive Innovation’, a change which was, in part, a recognition that disruption is a process and that it arises from ground-level practice more than from the intrinsic features and design of technologies. Disruptive Innovation is also interested in the specific jobs that technologies do for users.

Disruptive Innovations disrupt by appealing to the low end of existing markets or by creating new markets. The transistor radio, on its introduction in the 1950s, was technologically inferior to valve-based radios but was affordable to a new constituency, teenagers. Having established a market niche, the transistor radio was able to improve its offering along sustaining innovation lines, eventually replacing the incumbent, achieving Disruptive Innovation (Christensen & Raynor 2003). The transistor radio was then the dominant player in the market until it was disrupted by radio over the internet, in a similar pattern.

Disruptive technologies also succeed by enabling users to get jobs done by means more convenient than the existing solution (Christensen & Raynor 2003). Wikipedia is not always approved in academia (Di Lauro & Johinke 2017; Leitch 2014) but gets used frequently (Kim et al. 2014; Selwyn & Gorard 2016) in preference to printed sources of synoptic information because it is free, convenient and easy to use. Its success may be measured in part by the decision of Encyclopaedia Britannica to stop print publication in 2012.

Latterly, a third category of Efficiency Innovation was introduced (Christensen et al. 2016). Efficiency Innovations do not necessarily comprise an enhancement but they effect cost savings, such as self-service tills in supermarkets which remove the need for checkout staff. Efficiency Innovations pose a threat to jobs but they also enable jobs to be done with fewer resources. The self-checkout terminals for books in university libraries are an example of Efficiency Innovation in higher education.

The core theory of Disruptive Innovation has been developed and critiqued. Some writers have pointed out that Disruptive Innovation can begin at the high end of markets: Carr (2005) cites FedEx, which offered a premier document delivery service to high-end customers. Having established a market position, the FedEx was able to spread down through the rest of the market. Another example of successful high-end disruption is the calculator: ‘The calculator’s attributes were exactly what high-end customers… wanted, and these high-end customers were the first to buy. As calculator prices came down, calculators relatively quickly diffused down-market’ (Schmidt & Druehl 2008, p. 362). An advantage of a high-end disruption is that it tends to be visible from the outset and can create a product, service or brand to which mainstream customers aspire (Flavin 2020).

Disruptive Innovation is not necessarily good: Cortez (2014) cites the case of financial instruments, which allowed widespread, unregulated participation in financial markets and which contributed significantly to the crash in 2008. Moreover, Birkin and Polesie (2011) argue, ‘Technology employed in stock markets means that major amounts of money are transferred and invested and hence intervene in the world on the basis of increasingly abstract mathematical relationships’ (p. 249). Disruptive Innovation disrupts, but it does not improve as a matter of course.

The fiercest criticism of Disruptive Innovation has come from another Harvard professor, Jill Lepore (2014). In an article in the New Yorker, she honed-in on a frequent methodology used for Disruptive Innovation studies, the retrospective case study, which allows for the cherry-picking of instances which validate the theory. Furthermore, Disruptive Innovation has not enjoyed conspicuous success as a predictive theory: Christensen cofounded the Disruptive Growth Fund in 2000 to support Disruptive Innovations but it closed within a year, having lost nearly two-thirds of its value (Danneels 2006). Furthermore, Christensen predicted that the Apple iPhone would not be successful, seeing it as a sustaining innovation on the cell phone rather than as a Disruptive Innovation, which would become people’s primary means of connecting to the internet (McGregor 2007). However, Disruptive Innovation remains a useful lens for analysing the function of technologies in higher education, and why some succeed and some fail (Flavin 2017). By making goods and services quick, convenient and cheap, Disruptive Innovations appeal to specific market segments and frequently grow from their initial niche. A series of markets have been disrupted, including air travel through the emergence of budget airlines (Kumar 2006), the car production industry (Christensen et al. 2009; Schmidt & Druehl 2008), the car rental industry (Markides & Sosa 2013) and digital photography (Benner & Tripsas 2012). However, higher education has remained largely unchanged (Marginson 2013). Technologies have supported existing forms of learning, teaching and assessment, not transformed them. That said, technology retains the potential to disrupt, and if technology changes pedagogy, assessment is unlikely to remain unchanged indefinitely.

Having summarised Disruptive Innovation theory, the next stage of the article summarises the different levels of data gathering, to inform and substantiate the discussion.

Materials and method

Assessment on the PGLA is incremental: in four sequential tasks, students start by writing short reflections on class content and end up writing a strategy to address a target from the United Nations Sustainable Development Goals (United Nations no date), creating an e-portfolio on global leadership. The e-portfolio is an established form of assessment, which started out being used in higher education for career-related purposes (Chatham-Carpenter et al. 2010). E-portfolios can be defined as, ‘digitally mediated, learner-centred, deliberate collections of work aimed at embodying sophisticated achievement’ (Deneen et al. 2018, p. 488). Furthermore, Bowman et al. (2016) argue e-portfolios encourage, ‘reflection as an iterative process’ (p. 8). A study by Ciesielkiewicz (2019) showed that trainee teachers at a university in Spain recognise the value of e-portfolios for career advancement, but e-portfolios may be less well suited as an assessment vehicle for a course which is not explicitly vocational. E-portfolios are used on the PGLA to encourage reflection: in a sub-section of the final assessment, students are asked to revisit their opening thoughts and feelings about leadership (which they submit on their application forms) and consider how their views may have changed.

In the first year of the PGLA, students were given the option of submitting their assignments as word-processed documents or blog entries. Only 5% of students took the blogging option. In years 2 and 3 (2016–17 and 2017–18), the Mahara app was used for assessment. Mahara is an e-portfolio system. It is free and open-source and describes itself as, ‘a form of Personal Learning Environment’ (Mahara 2019). Users can store a range of digital artefacts on Mahara, including text, audio and video files, creating an e-portfolio. Mahara has a social networking facility, too, as other users can see the sources collated by the primary user, as long as that user has given permission.

The decision to switch to Mahara was made because it had been used successfully in another interdisciplinary course at King’s, with one student making extensive use of audio files to creatively reproduce the hearing voices symptom in schizophrenia. The work was subsequently the source of an article published in an academic journal (Flavin & James 2018). On the PGLA, a 3-min video, produced by an in-house learning technologist, showed students how to upload material to Mahara. Face-to-face sessions on using Mahara were also available, though optional. Tse et al. (2018) argue that students’ approach to e-portfolios is partly determined by lecturers’ knowledge of, confidence in and educational commitment to the technology: competent training and induction was available for the use of Mahara on the PGLA but was targeted primarily at students.

In 2018–19 and 2019–20, the in-built blogging facility on the Moodle VLE was used, owing to underuse of the range of capabilities on Mahara. Both the external blogging apps, available in the first iteration of the course, and Mahara, used for 2 consecutive years, enabled the use of multi-media, but the vast majority of students (89.6%) did not use these affordances and treated the apps like word-processing software, as they have in subsequent years with the blogging facility on the Moodle VLE. Mueller and Bair (2018) reported on the use of e-portfolios on an interdisciplinary course at the University of Calgary, Canada (a tailored version of the WordPress, a blogging platform, was the specific technology used). They concluded, ‘There was little indication in the final ePortfolio assignments that students had been using their portfolios to document the development of their thinking, explore their own perspectives, or consider their learning on a metacognitive level’. Similarly, Roberts (2018), in a study of e-portfolios, found, ‘much of the reflective journal writing and assignment drafting was being completed in other platforms (predominately Microsoft Word). The students were then either uploading these files to their asset stores as evidence, or cutting and pasting the text from one platform to another’ (p. 320). E-portfolios can result in instrumental rather than creative or reflective submissions, serving efficiency rather than transformation.

Students on the PGLA undertake a course evaluation survey each year, which is used as a data source in this article. The data from year 1 (2015–16) were no longer available; hence, four end of course evaluation surveys are used as data, from 2016 to 2020. In addition, in 2017, a closed LinkedIn group was created, at a student’s suggestion, to enable ongoing networking opportunities for former PGLA students. The group has 106 present and former students as members. A survey on assessment for the PGLA was sent to all group members in July 2020, comprising a second level of data for this article. In total, 55 students submitted the survey (51.9% survey return, to one decimal place).

As a third and final level of data, the author conducted a focus group with six students in July 2020 to discuss the assessment on the 2019–20 iteration of the course, and learning and assessment practices more widely. Microsoft Team was used to facilitate the focus group. The group setting allowed students to respond to each other’s views. The students knew each other and experienced no evident problems in interacting. The focus group countered the lack of depth in aspects of the survey responses.

Disruptive Innovation is used in the article to analyse the annual evaluation surveys, the specific assessment survey and the focus group. The overall approach is qualitative content analysis with a directed approach. Qualitative content analysis with a directed approach is used in order to validate a theory and extend its applications (Hsieh & Shannon 2005, pp. 1281, 1283). Bryman (2016) describes qualitative content analysis as, ‘A searching-out of underlying themes in the materials being analysed’ (p. 563), which is an aim of this research, to draw out students’ thoughts and feelings about online assessment on a Leadership course. A directed approach to qualitative content analysis starts with a theory. In the case of this article, the theory of Disruptive Innovation provides the three categories of Disruptive Innovation, Sustaining Innovation and Efficiency Innovation.

Having outlined the research methodology and the sources of data, the next section describes the results from the various data sources.

Results

In the course evaluation survey for 2016–17, 13 students completed the survey, from a cohort of 20. In response to the question, ‘I am confident that I know what was required of me in the assessment’, 10 students responded ‘slightly agree’ and three students responded ‘slightly disagree’. In response to the question, ‘How easy did you find it to use the Mahara app for your assessment’, one student responded ‘very easy’, six students responded ‘quite easy’, five students responded ‘Neither difficult nor easy’ and one student responded ‘quite difficult’ (five responses were available, in a Likert scale format; see Figure 1).

Fig 1
Figure 1. How easy is using Mahara? (2016–17 cohort, n.13).

In 2017–18, the cohort size was extended to 40 (20 undergraduate and 20 postgraduate students), and the course evaluation survey also changed, asking students if they felt supported by the course team in undertaking the assessment. In the undergraduate cohort, eight out of 20 students responded. Seven stated they were confident that they knew what was required of them in the assessment and had sufficient support to complete the assessment (one student did not respond to these two questions). When asked if they found it easy to use the Mahara app, three students strongly agreed, two somewhat agreed, one somewhat disagreed and one strongly disagreed (one did not respond to the question).

For the 2017–18 postgraduate cohort, nine students responded to the survey. Four stated they were confident that they knew what was required of them in the assessment and had sufficient support to complete the assessment (five students did not respond to these two questions). When asked, if they found it easy to use the Mahara app, one student strongly agreed, one somewhat agreed, one neither agreed nor disagreed, and one somewhat disagreed (five did not respond to the question).

As shown in Figure 2, the majority of students in the cohort did not have a problem with the ease of use of Mahara, a general trend which had also been evident in 2016–17.

Fig 2
Figure 2. Mahara is easy to use (2017–18, n.11).

In 2018–19, when a decision had been made by the course team to use the blogging facility on the Moodle VLE because of low engagement with the multi-media capabilities of Mahara, three undergraduate students responded to the end of course evaluation survey, from a cohort of 20. All three stated they were confident that they knew what was required of them in the assessment and had sufficient support to complete the assessment. For the postgraduate cohort (n. 25, the cohort having expanded), nine students responded. All nine stated they were confident that they knew what was required of them in the assessment and had sufficient support to complete the assessment.

For the 2019–20 undergraduate cohort (n. 20), 13 students responded. Eleven students stated they were confident that they knew what was required of them in the assessment and had sufficient support to complete the assessment (two students did not respond to these two questions). For the postgraduate cohort (n. 26, owing to a further, marginal expansion), 13 students responded. Ten students stated they were confident that they knew what was required of them in the assessment and had sufficient support to complete the assessment (three students did not respond to these two questions).

Beyond the annual evaluation survey, a specific survey on assessment for the PGLA was distributed, via a closed LinkedIn group, to 106 current and former PGLA students, in July 2020. Fifty-five students responded. When asked, ‘Do you enjoy learning to use new apps or software?’, 38 students stated ‘Yes’, two stated ‘no’, 13 stated ‘No opinion’ and two did not reply. When asked, ‘Do you enjoy undertaking innovative forms of assessment?’, 48 students stated ‘Yes’, five stated ‘No’ and two did not reply.

When asked to evaluate factors influencing their usage of a new app for study purposes, using Likert scale responses, 19 students stated cost was ‘extremely important’, 17 ‘very important’, 11 ‘moderately important’, six ‘slightly important’ and one ‘not at all important’ (there were 54 responses to this and the subsequent question). Twenty-three stated convenience (how easy it is to access the app) was ‘extremely important’, 24 ‘very important’, six ‘moderately important’ and one ‘slightly important’. Twenty stated simplicity (how self-explanatory the app is) was ‘extremely important’, 22 ‘very important’, nine ‘moderately important’ and three ‘slightly important’. Twenty-two stated ease of use was ‘extremely important’, 21 ‘very important’, 10 ‘moderately important’ and one ‘slightly important’.

When asked to evaluate factors influencing their usage of a new app for social purposes, 24 stated cost was ‘extremely important’, 15 ‘very important’, nine ‘moderately important’, five ‘slightly important’ and one ‘not at all important’. Twenty-four stated convenience was ‘extremely important’, 24 ‘very important’, five ‘moderately important’ and one ‘slightly important’. Nineteen stated simplicity was ‘extremely important’, 19 ‘very important’, 13 ‘moderately important’, two ‘slightly important’ and one ‘not at all important’. Twenty-one stated ease of use was ‘extremely important’, 21 ‘very important’, 11 ‘moderately important’, and one ‘slightly important’.

When considering the core criteria for disruptive technologies (Christensen 1997, p. xv), students consistently rated them as ‘extremely important’ or ‘very important’. Cost and convenience were seen as more important for social rather than study purposes (Figures 3 and 4), and simplicity and ease of use were seen as more important for study than for social purposes (Figures 5 and 6), but all four criteria were seen as important by students who completed the assessment survey.

Fig 3
Figure 3. The importance of cost as a criterion for using a new app.

Fig 4
Figure 4. The importance of convenience as a criterion for using a new app.

Fig 5
Figure 5. The importance of simplicity as a criterion for using a new app.

Fig 6
Figure 6. The importance of ease of use as a criterion for using a new app.

The figures show very little difference when they come to convenience and ease of use, but more noticeable difference in respect of cost and simplicity. Students were more concerned with cost as a criterion when it came to using an app for social purposes, and more concerned with simplicity when it came to using an app for study purposes.

The final layer of data for this study, the focus group, took place online in July 2020, facilitated by the author, through Microsoft Teams. Six students attended. All were postgraduate and had taken the PGLA in 2019–20. Of the six, one had included audio-visual materials in work submitted for assessment; the rest had submitted word-processed documents. Thoughts and feelings about assessment on the PGLA were generally positive: ‘I like that it felt cumulative’. When asked about the more general purpose of assessment in higher education, responses were mixed and ranged from ‘Assessment is for you to reflect on what you’ve done so far’ to ‘Assessments kind of force you to reflect on what you’ve learned so far and then tie those ideas together’ to ‘Assessments are for your professors to evaluate you… for you to show off your knowledge’ and to ‘Assessment is very much something that helps universities… to really discern between applicants… it’s something quantifiable’.

One student said they appreciated ‘the freedom to choose the topic you want to learn about’, but another stated ‘I don’t like an overly creative assignment’. Further comments included, ‘There’s like a sweet spot, I like having a little choice or the option for some creativity but I think in a way I find it more limiting when you’re given an assignment where they want you to do something really off the beaten path’. When asked about their least favourite form of assessment, one student stated, ‘My heart always sinks slightly when it’s a new form of assessment’, and when asked about innovative forms of learning and assessment, students stated, ‘I like learning in innovative ways but I appreciate less innovative assessment’. One student added, ‘I like learning in a variety of ways and more innovative ways, and then I appreciate, like, less innovative assessment’ and ‘I’ve been groomed to do academic work in a certain way’.

Having summarised results from the different levels of data, the next section analyses the results through the lens of Disruptive Innovation theory, and in the context of other studies.

Discussion and conclusion

This study set out to analyse students’ thoughts and feelings about online assessment, using Disruptive Innovation theory as a lens through which to interpret the research findings. The PGLA is an interdisciplinary course, available to students across all nine faculties at King’s College London. It is innovative in the composition of its cohort (courses are more commonly contained within their respective faculties), yet assessment has gravitated towards traditional practices despite being located online, with the majority of students submitting word-processed reflections. Students have opportunities to be more innovative in work submitted for assessment but have tended to produce formally unadventurous work in an essay format.

In 2016–17 and 2017–18, Mahara was a new technology on the course but few students used its capability to incorporate audio-visual material. Instead, the new technology was used to facilitate a traditional assessment. The underusage of the audio-visual features on Mahara was noticeable over the 2 years it was used: only 10.4% of students used the additional features. Deneen (2013) also reported negative responses to the use of Mahara from both teachers and students at a university in Hong Kong, especially in relation to the learning curve required to use the technology. Data from the present study indicate the learning curve to use Mahara was not unduly steep, but that usage of the affordances of the medium was, in practice, very limited. It is noteworthy that Mahara had had some success on another interdisciplinary course at King’s, with one student creatively reproducing the hearing voices symptom in schizophrenia (Flavin & James 2018). The interdisciplinary course in question examined links between mental health, mental illness and creative practice. It is possible that the subject area lent itself more readily to creative use of the medium. A leadership course taught from a social sciences perspective may not have been perceived by students as a similarly creative learning environment. Disruptive Innovation argues successful new technologies are simpler, more convenient and easier to use than the incumbents they threaten. When new technologies are more complex, they are less likely to succeed. When new technologies are not complex, usage is directed towards getting jobs done by simple and convenient means (Christensen & Raynor 2003). Using Mahara did not pose a significant technological challenge, but students did not make full use of Mahara.

The majority of respondents to the assessment survey sent via LinkedIn stated they enjoyed innovative forms of assessment, but their practices were more likely to be traditional, eschewing the innovative possibilities of the technologies made available to them. Cost was a more important criterion for social apps than for study apps, perhaps because the cost was more likely to be directly borne by the student. Simplicity was a more important criterion for a study app than a social app, perhaps articulating a jobs-to-be-done attitude towards assessment (Christensen & Raynor 2003, p. 79). The criteria commonly associated with technology usage from a Disruptive Innovation perspective – cost, convenience, simplicity and ease of use (Christensen 1997) – were important to students, but it did not follow that they used the technologies themselves in obviously innovative ways. A declared enthusiasm for innovative assessment was not substantiated by innovative practice in assessment. It is possible that innovative online assessment is unlikely to be successful if operated in isolation: Roberts (2018) argues, of e-portfolios, ‘the process needed to be embedded across the degree with constant and varied opportunities to use the platforms. This would appear to result in a streamlining of the process for students and make it more efficient’ (p. 321). Innovative online assessment is more likely to work in the context of a strategic and institutional commitment, but it is unclear whether universities will risk reputation and revenue by proactively undertaking a pedagogical disruption from the top down. Studies of university technology-enhanced learning strategies from a Disruptive Innovation perspective showed strategies to be conservative, featuring a surface commitment to innovation, which, on analysis, was shown to be Sustaining Innovation or Efficiency Innovation, rather than Disruptive Innovation(Flavin & Quintero 2018, 2020), pursuing modest advancement along an existing trajectory, or offering the more efficient usage of existing resources. Universities have the option of strategically enacting high-end disruption in learning, teaching and assessment but have yet to do so. High-end disruption could create a pedagogical and assessment template, underpinned by technology, which the rest of the sector could emulate.

Students opt for simple, convenient and easy to use technologies, in line with Disruptive Innovation theory. Brown (2015) argues, in relation to e-portfolios, ‘To choose the best platform, the first consideration should be the tool’s affordances; then, three other characteristics – usability, accessibility, and cost/help – must align with and support those affordances’ (p. 338). Although Brown’s article is not written from a Disruptive Innovation perspective, it effectively endorses a Disruptive Innovation analysis: ensuring technologies are simple and convenient increases the likelihood of student engagement: steep upward learning curves provoke steep downward engagement. The changes in mode of online assessment in this case study have been driven by students’ practices, but they have driven away from innovative assessment.

Students’ understandings of assessment and its role shape their usage of technologies. Deneen et al. (2018) argue, ‘In eportfolio adoption and use, it matters how students understand assessment, how they relate to technology and how those conceptions interact’ (p. 494). Focus group participants in the present study showed some awareness of the intrinsic value of assessment, but a greater sense of the extrinsic, instrumental purpose of assessment, to measure and classify. The technological forms of assessment used on the PGLA did not pose substantial problems, but a more fundamental understanding of assessment and its purposes militated against innovative practice. As Roberts (2018) argues, ‘If students are to submit their work for university assessment, it is no longer a personal reflection and there is pressure to write in a specific way with standardised content that will meet external requirements and therefore receive a good mark’ (p. 314).

Students identify the core job of completing the task and the simplest and easiest means of getting the job done, which is often a continuum of the means by which they have always completed assessed tasks; one student in the focus group discussed being, ‘groomed to do academic work in a certain way’. Students are undertaking online assessment but are not demanding innovative online assessment methodologies. The impediments to innovative online assessment are not technological, but changes wrought by technological innovation have yet to feed through into assessment practice.

Brown (2015) suggests students expect technologies for learning, specifically e-portfolios, to be as simple and convenient as technologies for commerce: ‘when making a purchase online, the steps to clicking BUY are obvious and as frictionless as possible. The same fluid trajectory is anticipated in ePortfolio building’ (p. 337). When learning technologies are not similarly frictionless, resistance is likely to occur. However, learning is a more complex process than making a purchase. Part of the challenge for educators is to explicate the learning process, such that the technological barriers to online assessment are absent or smoothed, enabling a more expansive approach on the part of students, who can concentrate on learning rather than technology.

The limitations of this study include the point that students’ evaluations are not always reliable; evaluations capture a snapshot, but students may give different responses at different times, and some students were being asked, in the one-off assessment survey, to reflect on an assessment they undertook 5 years previously. That said, different forms of data were used in this study, with consistent themes emerging. A further limitation of the study is that it does not analyse lecturers’ feedback on online assessments submitted by students, but this is beyond the scope of the present study, which is about the students’ opinions of online assessments, and factors underpinning their assessment practices. Finally, as this is a case study focusing on one course at one university, it is not possible to say that the findings will be replicated in another institutional or subject context.

This study adds to our understanding by applying Disruptive Innovation theory to online assessment. It shows students identifying with the criteria for Disruptive Innovation in their usage of technology, but not deploying technology itself in innovative ways. Students, this study suggests, are willing to be formally innovative in their online assessment choices, but at the same time, they are pedagogically conservative. Technology is used to make the assessment process more efficient but it is not being used to disrupt existing pedagogical norms. If universities are to transform online assessment, it may need to be through a strategic commitment.

The development of online assessment in higher education provides more of a pedagogical than technological challenge, a challenge which may need to be addressed on an institutional and possibly sectorial level. Traditional perspectives on assessment have, to date, survived digitally led pedagogical evolution, in part because higher education has tended to accommodate technology rather than being transformed by it. New perspectives on online assessment are needed, linked to pedagogy and strategy, and informed by theory.

References

Benner, M. J. & Tripsas, M. (2012) ‘The influence of prior industry affiliation on framing in nascent industries: the evolution of digital cameras’, Strategic Management Journal, vol. 33, no. 3, pp. 277–302. doi: 10.1002/smj.950

Birkin, F. & Polesie, T. (2011) ‘An epistemic analysis of (un) sustainable business’, Journal of Business Ethics, vol. 103, no. 2, pp. 239–253. doi: 10.1007/s10551-011-0863-4

Bowman, J., et al., (2016) ‘The use of ePortfolios to support metacognitive practice in a first-year writing program’, International Journal of ePortfolio, vol. 6, no. 1, pp. 1–22.

Brown, S. (2015), ‘The impact of the ePortfolio tool on the process: functional decisions of a new genre’, Theory into Practice, vol. 54, no. 4, pp. 335–342. doi: 10.1080/00405841.2015.1077618

Bryman, A. (2016) Social Research Methods, 5th edn, Oxford University Press, Oxford.

Carr, N. G. (2005) ‘Top-down disruption’, Strategy and Business, vol. 39, [online] Available at: https://www.strategy-business.com/article/05203?gko=81cf4

Chatham-Carpenter, A., Seawel, L. & Raschig, J. (2010) ‘Avoiding the pitfalls: current practices and recommendations for ePortfolios in higher education’, Journal of Educational Technology Systems, vol. 38, no. 4. pp. 437–456. doi: 10.2190/ET.38.4.e

Christensen, C. M. (1997) The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail, Harvard Business School Press, Boston, MA.

Christensen, C. M., Bartman, T. & van Bever, D. (2016) ‘The hard truth about business model innovation’, MIT Sloan Management Review, [online] Available at: http://sloanreview.mit.edu/paper/the-hard-truth-about-business-model-innovation/

Christensen, C. M. & Eyring, H. J. (2011) The Innovative University: Changing the DNA of Higher Education from the Inside Out, Jossey-Bass, San Francisco, CA.

Christensen, C. M., Grossman, J. & Hwang, J. (2009) The Innovator’s Prescription: A Disruptive Solution for Health Care, McGraw-Hill, New York, NY.

Christensen, C. M. & Raynor, M. E. (2003) The Innovator’s Solution: Creating and Sustaining Successful Growth, Harvard Business School Press, Boston, MA.

Ciesielkiewicz, M. (2019) ‘The use of e-portfolios in higher education: from the students’ perspective’, Issues in Educational Research, vol. 29, no. 3, pp. 649–667, [online] Available at: http://www.iier.org.au/iier29/ciesielkiewicz.pdf

Cortez, N. (2014) ‘Regulating disruptive innovation’, Berkeley Technology Law Journal, vol. 29, no. 1, pp. 175–228, [online] Available at: https://scholar.smu.edu/cgi/viewcontent.cgi?article=1003&context=law_faculty

Danneels, E. (2006) ‘From the guest editor: dialogue on the effects of disruptive technology on firms and industries’, The Journal of Product Innovation Management, vol. 23, pp. 2–4. doi: 10.1111/j.1540-5885.2005.00174.x

Deneen, C. C. (2013) ‘Eportfolios in a higher education context: preliminary findings on assessment and technology issues’, Journal of Information Systems Technology & Planning, vol. 6, no. 17, pp. 145–160.

Deneen, C. C., Brown, G. T. L. & Carless, D. (2018) ‘Students’ conceptions of eportfolios as assessment and technology’, Innovations in Education and Teaching International, vol. 55, no. 4, pp. 487–496. doi: 10.1080/14703297.2017.1281752

Di Lauro, F. & Johinke, R. (2017) ‘Employing Wikipedia for good not evil: innovative approaches to collaborative writing assessment’, Assessment & Evaluation in Higher Education, vol. 42, no. 3, pp. 478–491. doi: 10.1080/02602938.2015.1127322

Flavin, M. (2017) Disruptive Technology Enhanced Learning: The use and misuse of digital technologies in higher education, Palgrave Macmillan, London.

Flavin, M. (2020) Re-imagining Technology Enhanced learning: Critical perspectives on Disruptive Innovation, Palgrave Macmillan, London.

Flavin, M. & James, B. (2018) ‘To give an outsider an idea of what it could be like’: A case study of the creative representation of hearing voices’, Arts and Humanities in Higher Education, vol. 17, no. 1, pp. 134–147.

Flavin, M. & Quintero, V. (2018) ‘UK higher education institutions’ technology-enhanced learning strategies from the perspective of disruptive innovation’, Research in Learning Technology, vol. 26. doi: 10.25304/rlt.v26.1987

Flavin, M. & Quintero, V. (2020) ‘An international study of technology enhanced learning-related strategies from the perspective of disruptive innovation’, Interactive Technology and Smart Education, vol. 17, no. 4, pp. 475–488.

Hsieh, H.-F. & Shannon, S. E. (2005) ‘Three approaches to qualitative content analysis’, Qualitative Health Research, vol. 15, no. 9, pp. 1277–1288. doi: 10.1177/1049732305276687

Kim, K. S., Sin, S.-C. J. and Tsai, T.-I. (2014) ‘Individual differences in social media use for information seeking’, The Journal of Academic Librarianship, vol. 40, no. 2, pp. 171–178. doi: 10.1016/j.acalib.2014.03.001

Kumar, N. (2006) ‘Strategies to fight low-cost rivals’, Harvard Business Review, vol. 84, no.12, pp. 104–112.

Leitch, T. (2014) Knowledge, Authority and Liberal Education in the Digital Age, Johns Hopkins University Press, Baltimore, MD.

Lepore, J. (2014) ‘The disruption machine: what the gospel of innovation gets wrong’, The New Yorker, vol. 90, no. 17, pp. 30–36.

Mahara. (2019) About Mahara, [online] Available at: https://mahara.org/view/view.php?id=2

Marginson, S. (2013) ‘The impossibility of capitalist markets in higher education’, Journal of Education Policy, vol. 28, no. 3, pp. 353–370. doi: 10.1080/02680939.2012.747109

Markides, C. & Sosa, L. (2013) ‘Pioneering and first mover advantages: the importance of business models’, Long Range Planning, vol. 46, pp. 325–334. doi: 10.1016/j.lrp.2013.06.002

McGregor, J. (2007) ‘Clayton Christensen’s innovation brain’, Bloomberg, [online] Available at: https://www.bloomberg.com/news/articles/2007-06-15/clayton-christensens-innovation-brainbusinessweek-business-news-stock-market-and-financial-advice

Mueller, R. A. & Bair, H. (2018) ‘Deconstructing the notion of ePortfolio as a “high impact practice”: a self-study and comparative analysis’, Canadian Journal for the Scholarship of Teaching and Learning, vol. 9, no. 3, [online] Available at: https://files.eric.ed.gov/fulltext/EJ1203557.pdf

Roberts, P. (2018) ‘Developing reflection through an ePortfolio-based learning environment: design principles for further implementation’, Technology, Pedagogy and Education, vol. 27, no. 3, pp. 313–326. doi: 10.1080/1475939X.2018.1447989

Schmidt, G. M. & Druehl, C. T. (2008) ‘When is a disruptive innovation disruptive?’, The Journal of Product Innovation and Management, vol. 25, pp. 347–369. doi: 10.1111/j.1540-5885.2008.00306.x

Selwyn, N. & Gorard, S. (2016) ‘Students’ use of Wikipedia as an academic resource – patterns of use and perceptions of usefulness’, The Internet and Higher Education, vol. 28, pp. 28–34. doi: 10.1016/j.iheduc.2015.08.004

Tse, C. T., Scholz, K. W. & Lithgow, K. (2018) ‘Beliefs or intentionality? Instructor approaches to ePortfolio pedagogy’, The Canadian Journal for the Scholarship of Teaching and Learning, vol. 9, no. 3. doi: 10.5206/cjsotl-rcacea.2018.3.10

United Nations. (no date). Sustainable Development Goals, [online] Available at: https://sustainabledevelopment.un.org/?menu=1300