Migration and transformation: a sociomaterial analysis of practitioners’ experiences with online exams

  • Stuart Allan Director of Online Learning, Edinburgh Business School, Heriot-Watt University, Edinburgh, Scotland
Keywords: assessment, digital exams, online assessment, online exams, sociomateriality

Abstract

Many institutions are making the move from pen and paper to online examinations, but the literature offers relatively few critical reflections on the ramifications of such a shift. This research presents evidence of the ways in which the social and human practices of online exams are deeply entangled with the material and technological, and cautions against the reinscribing of essentialist or instrumentalist assumptions about technology in assessment practices. Through semi-structured interviews with eight practitioners in Norway, the Netherlands, the UK and Ireland, it analyses the impact, dimensions and limitations of two main discourses: migration, whereby exam technologies are assumed to be neutral instruments used independently by humans to realise their preordained intentions; and transformation, whereby the essential and inalienable qualities of technologies can be released to ‘transform’ or ‘enhance’ assessment. Its findings indicate that: (1) exam technologies are neither inherently neutral nor essentially transformational; (2) implementation projects underpinned by the migration discourse can be much more complex and resource-intensive than anticipated; and (3) ‘transformative’ change may be value-laden and driven by assumptions. Given the complex and entangled nature of online exams, practitioners are encouraged to think creatively about how assessment strategies align with educational goals, to consider the limitations of current discourses and to analyse critically the relational and performative roles of digital technologies.

Downloads

Download data is not yet available.

References

References



Barad, K. (2003) ‘Posthumanist performativity: toward an understanding of how matter comes to matter’, Signs, vol. 28, pp. 801–831. doi: 10.1086/345321


Bayne, S. (2015) ‘What’s the matter with ‘technology-enhanced learning?’, Learning, Media and Technology, vol. 40, pp. 5–20. doi: 10.1080/17439884.2014.915851


Berggren, B., Fili, A. & Nordberg, O. (2015) ‘Digital examination in higher education – experiences from three different perspectives’, International Journal of Education and Development Using Information and Communication Technology, vol. 11, pp. 100–108.


Boud, D. & Soler, R. (2016) ‘Sustainable assessment revisited’, Assessment and Evaluation in Higher Education, vol. 41, pp. 400–413. doi: 10.1080/02602938.2015.1018133


Carless, D. (2007) ‘Learning-oriented assessment: conceptual bases and practical implications’, Innovations in Education and Teaching International, vol. 44, pp. 57–66. doi: 10.1080/14703290601081332


Cramp, J., et al., (2019) ‘Lessons learned from implementing remotely invigilated online exams’, Journal of University Teaching and Learning Practice, vol. 16, no. 1., [online] Available at: https://ro.uow.edu.au/jutlp/vol16/iss1/10


Crisp, G., Guàrdia, L. & Hillier, M. (2016) ‘Using e-assessment to enhance student learning and evidence learning outcomes’, International Journal of Educational Technology in Higher Education, vol. 13. doi: 10.1186/s41239-016-0020-3


D’Souza, K. A. & Siegfeldt, D. V. (2017) ‘A conceptual framework for detecting cheating in online and take-home exams’, Decision Sciences: Journal of Innovative Education, vol. 15, pp. 370–391. doi: 10.1111/dsji.12140


Escudier, M. P., et al., (2011) ‘University students’ attainment and perceptions of computer-based and traditional tests in a high-stakes examination’, Journal of Computer Assisted Learning, vol. 27, pp. 440–447. doi: 10.1111/j.1365-2729.2011.00409.x


Fenwick, T., Edwards, R. & Sawchuk, P. (2011) Emerging Approaches to Educational Research: Tracing the Sociomaterial. Routledge, London.


Ferrell, G. (2014) Electronic Management of Assessment (EMA): A Landscape Review, JISC, Bristol, [online] Available at: http://www.eunis.org/wp-content/uploads/2015/05/EMA_REPORT.pdf


Gibbs, G. & Simpson, C. (2004) ‘Conditions under which assessment supports students’ learning’, Learning and Teaching in Higher Education, vol. 1, pp. 3–33, [online] Available at: http://eprints.glos.ac.uk/3609/


Gillespie, T. (2010) ‘The politics of ‘platforms”’, New Media and Society, vol. 12, pp. 347–364. doi: 10.1177/1461444809342738


Glaser, B. B. & Strauss, A. L. (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research, Transaction, New Brunswick, NJ.


Hamilton, E. & Friesen, N. (2013) ‘Online education: a science and technology studies perspective/Éducation en ligne: perspective des études en science et technologie’, Canadian Journal of Learning and Technology/La Revue Canadienne de l’Apprentissage et de la Technologie, vol. 39, [online] Available at: https://www.learntechlib.org/p/54417/. doi: 10.21432/T2001C


Hannon, J. (2013) ‘Incommensurate practices: sociomaterial entanglements of learning technology implementation’, Journal of Computer Assisted Learning, vol. 29, pp. 168–178. doi: 10.1111/j.1365-2729.2012.00480.x


Hepplestone, S., et al., (2011) ‘Using technology to encourage student engagement with feedback: a literature review’, Research in Learning Technology, vol. 19, pp. 117–127. doi: 10.3402/rlt.v19i2.10347


Hillier, M. & Fluck, A. (2015) ‘A pedagogical end game for exams: a look 10 years into the future of high stakes assessment’, in Proceedings of the Australasian Society for Computers in Learning in Tertiary Education (Ascilite), eds T. Reiners et al., Perth, Australia, 29 November – 2 December 2015, pp. 463–470, [online] Available at: http://www.2015conference.ascilite.org/wp-content/uploads/2015/11/ascilite-2015-proceedings.pdf


Hylton, K., Levy, Y. & Dringus, L. P. (2016) ‘Utilizing webcam-based proctoring to deter misconduct in online exams’, Computers and Education, vol. 92–93, pp. 53–63. doi: 10.1016/j.compedu.2015.10.002


James, R. (2016) ‘Tertiary student attitudes to invigilated, online summative examinations’, International Journal of Educational Technology in Higher Education, vol. 13. doi: 10.1186/s41239-016-0015-0


Johnston, B., MacNeill, S. & Smyth, K. (2018) Conceptualising the Digital University: The Intersection of Policy, Pedagogy and Practice. Palgrave Macmillan, London.


Joint Information Systems Committee (JISC) (2018) Designing Learning and Assessment in a Digital Age. JISC, Bristol, [online] Available at: https://www.jisc.ac.uk/guides/designing-learning-and-assessment-in-a-digital-age


Karaman, S. (2011) ‘Examining the effects of flexible online exams on students’ engagement in e-learning’, Educational Research and Reviews, vol. 6, pp. 259–264.


Karim, N. A. & Shukur, Z. (2016) ‘Proposed features of an online examination interface design and its optimal values’, Computers in Human Behaviour, vol. 64, pp. 414–422. doi: 10.1016/j.chb.2016.07.013


Khare, A. & Lam, H. (2008) ‘Assessing student achievement and progress with online examinations: some pedagogical and technical issues’, International Journal on E-learning, vol. 7, pp. 383–402.


Marshall, S. (2010) ‘Change, technology and higher education: are universities capable of organisational change?’, ALT-J, Research in Learning Technology, vol. 18, pp. 179–192. doi: 10.1080/09687769.2010.529107


Milone, A. S., et al., (2017) ‘The impact of proctored online exams on the educational experience’, Currents in Pharmacy Teaching and Learning, vol. 9, pp. 108–114. doi: 10.1016/j.cptl.2016.08.037


Myyry, L. & Joutsenvirta, T. (2015) ‘Open-book, open-web online examinations: developing examination practices to support university students’ learning and self-efficacy’, Active Learning in Higher Education, vol. 16, pp. 119–132. doi: 10.1177/1469787415574053


Newhouse, C. P. (2011) ‘Using IT to assess IT: towards greater authenticity in summative performance assessment’, Computers and Education, vol. 56, pp. 388–402. doi: 10.1016/j.compedu.2010.08.023


O’Keeffe, C. (2016) ‘Producing data through e-assessment: a trace ethnographic investigation into e-assessment events’, European Educational Research Journal, vol. 15, pp. 99–116. doi: 10.1177/1474904115612486


Orlikowski, W. (2010) ‘The sociomateriality of organisational life: considering technology in management research’, Cambridge Journal of Economics, vol. 34, pp. 125–141. doi: 10.1093/cje/bep058


Orlikowski, W. J. & Scott, S. V. (2008) ‘Sociomateriality: challenging the separation of technology, work and organization’, Academy of Management Annals, vol. 2, no. 1, pp. 433–474. doi: 10.1080/19416520802211644


O’Shea, C. & Fawns, T. (2014) ‘Disruptions and dialogues: supporting collaborative connoisseurship in digital environments’, in Advances and Innovations in University Assessment and Feedback, eds C. Kreber et al., Edinburgh University Press, Edinburgh, pp. 259–273.


Ripley, M. (2009) ‘Transformational computer-based testing’, in The Transition to Computer-based Assessment, eds F. Scheuermann & J. Bjornsson, European Commission Joint Research Centre, Ispra, Italy, pp. 92–98.


Rust, C. (2007) ‘Towards a scholarship of assessment’, Assessment and Evaluation in Higher Education, vol. 32, pp. 229–237. doi: 10.1080/02602930600805192


Schmidt, S. M. P., Ralph, D. L. & Buskirk, B. (2009) ‘Utilizing online exams: a case study’, Journal of College Teaching and Learning, vol. 6, pp. 1–8. doi: 10.19030/tlc.v6i8.1108


Tracy, S. J. (2013) Qualitative Research Methods: Collecting Evidence, Crafting Analysis, Communicating Impact, Blackwell Publishing, Oxford.


Ullah, A., Ziao, H. & Barker, T. (2019) ‘A study into the usability and security implications of text and image based challenge questions in the context of online examination’, Education and Information Technologies, vol. 24, pp. 13–39. doi: 10.1007/s10639-018-9758-7


Williams, J. B. & Wong, A. (2009) ‘The efficacy of final exams: a comparative study of closed-book, invigilated exams and open-book, open-web exams’, British Journal of Educational Technology, vol. 40, pp. 227–236. doi: 10.1111/j.1467-8535.2008.00929.x

Footnotes



1 ‘Online exams’ are understood here as high-stakes summative assessment events, mediated by digital technologies, often taking place in a defined place or time and under secure conditions (e.g. invigilation, restrictions on access to course materials, notes or communication).


2 Myyry and Joutsenvirta (2015) acknowledged that ‘the specific context and tradition of university examinations in Finland’ (p. 129) might have influenced their results.
Published
2020-01-27
How to Cite
AllanS. (2020). Migration and transformation: a sociomaterial analysis of practitioners’ experiences with online exams. Research in Learning Technology, 28. https://doi.org/10.25304/rlt.v28.2279
Section
Original Research Articles