Migration and transformation: a sociomaterial analysis of practitioners’ experiences with online exams
Abstract
Many institutions are making the move from pen and paper to online examinations, but the literature offers relatively few critical reflections on the ramifications of such a shift. This research presents evidence of the ways in which the social and human practices of online exams are deeply entangled with the material and technological, and cautions against the reinscribing of essentialist or instrumentalist assumptions about technology in assessment practices. Through semi-structured interviews with eight practitioners in Norway, the Netherlands, the UK and Ireland, it analyses the impact, dimensions and limitations of two main discourses: migration, whereby exam technologies are assumed to be neutral instruments used independently by humans to realise their preordained intentions; and transformation, whereby the essential and inalienable qualities of technologies can be released to ‘transform’ or ‘enhance’ assessment. Its findings indicate that: (1) exam technologies are neither inherently neutral nor essentially transformational; (2) implementation projects underpinned by the migration discourse can be much more complex and resource-intensive than anticipated; and (3) ‘transformative’ change may be value-laden and driven by assumptions. Given the complex and entangled nature of online exams, practitioners are encouraged to think creatively about how assessment strategies align with educational goals, to consider the limitations of current discourses and to analyse critically the relational and performative roles of digital technologies.
Downloads
References
References
Barad, K. (2003) ‘Posthumanist performativity: toward an understanding of how matter comes to matter’, Signs, vol. 28, pp. 801–831. doi: 10.1086/345321
Bayne, S. (2015) ‘What’s the matter with ‘technology-enhanced learning?’, Learning, Media and Technology, vol. 40, pp. 5–20. doi: 10.1080/17439884.2014.915851
Berggren, B., Fili, A. & Nordberg, O. (2015) ‘Digital examination in higher education – experiences from three different perspectives’, International Journal of Education and Development Using Information and Communication Technology, vol. 11, pp. 100–108.
Boud, D. & Soler, R. (2016) ‘Sustainable assessment revisited’, Assessment and Evaluation in Higher Education, vol. 41, pp. 400–413. doi: 10.1080/02602938.2015.1018133
Carless, D. (2007) ‘Learning-oriented assessment: conceptual bases and practical implications’, Innovations in Education and Teaching International, vol. 44, pp. 57–66. doi: 10.1080/14703290601081332
Cramp, J., et al., (2019) ‘Lessons learned from implementing remotely invigilated online exams’, Journal of University Teaching and Learning Practice, vol. 16, no. 1., [online] Available at: https://ro.uow.edu.au/jutlp/vol16/iss1/10
Crisp, G., Guàrdia, L. & Hillier, M. (2016) ‘Using e-assessment to enhance student learning and evidence learning outcomes’, International Journal of Educational Technology in Higher Education, vol. 13. doi: 10.1186/s41239-016-0020-3
D’Souza, K. A. & Siegfeldt, D. V. (2017) ‘A conceptual framework for detecting cheating in online and take-home exams’, Decision Sciences: Journal of Innovative Education, vol. 15, pp. 370–391. doi: 10.1111/dsji.12140
Escudier, M. P., et al., (2011) ‘University students’ attainment and perceptions of computer-based and traditional tests in a high-stakes examination’, Journal of Computer Assisted Learning, vol. 27, pp. 440–447. doi: 10.1111/j.1365-2729.2011.00409.x
Fenwick, T., Edwards, R. & Sawchuk, P. (2011) Emerging Approaches to Educational Research: Tracing the Sociomaterial. Routledge, London.
Ferrell, G. (2014) Electronic Management of Assessment (EMA): A Landscape Review, JISC, Bristol, [online] Available at: http://www.eunis.org/wp-content/uploads/2015/05/EMA_REPORT.pdf
Gibbs, G. & Simpson, C. (2004) ‘Conditions under which assessment supports students’ learning’, Learning and Teaching in Higher Education, vol. 1, pp. 3–33, [online] Available at: http://eprints.glos.ac.uk/3609/
Gillespie, T. (2010) ‘The politics of ‘platforms”’, New Media and Society, vol. 12, pp. 347–364. doi: 10.1177/1461444809342738
Glaser, B. B. & Strauss, A. L. (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research, Transaction, New Brunswick, NJ.
Hamilton, E. & Friesen, N. (2013) ‘Online education: a science and technology studies perspective/Éducation en ligne: perspective des études en science et technologie’, Canadian Journal of Learning and Technology/La Revue Canadienne de l’Apprentissage et de la Technologie, vol. 39, [online] Available at: https://www.learntechlib.org/p/54417/. doi: 10.21432/T2001C
Hannon, J. (2013) ‘Incommensurate practices: sociomaterial entanglements of learning technology implementation’, Journal of Computer Assisted Learning, vol. 29, pp. 168–178. doi: 10.1111/j.1365-2729.2012.00480.x
Hepplestone, S., et al., (2011) ‘Using technology to encourage student engagement with feedback: a literature review’, Research in Learning Technology, vol. 19, pp. 117–127. doi: 10.3402/rlt.v19i2.10347
Hillier, M. & Fluck, A. (2015) ‘A pedagogical end game for exams: a look 10 years into the future of high stakes assessment’, in Proceedings of the Australasian Society for Computers in Learning in Tertiary Education (Ascilite), eds T. Reiners et al., Perth, Australia, 29 November – 2 December 2015, pp. 463–470, [online] Available at: http://www.2015conference.ascilite.org/wp-content/uploads/2015/11/ascilite-2015-proceedings.pdf
Hylton, K., Levy, Y. & Dringus, L. P. (2016) ‘Utilizing webcam-based proctoring to deter misconduct in online exams’, Computers and Education, vol. 92–93, pp. 53–63. doi: 10.1016/j.compedu.2015.10.002
James, R. (2016) ‘Tertiary student attitudes to invigilated, online summative examinations’, International Journal of Educational Technology in Higher Education, vol. 13. doi: 10.1186/s41239-016-0015-0
Johnston, B., MacNeill, S. & Smyth, K. (2018) Conceptualising the Digital University: The Intersection of Policy, Pedagogy and Practice. Palgrave Macmillan, London.
Joint Information Systems Committee (JISC) (2018) Designing Learning and Assessment in a Digital Age. JISC, Bristol, [online] Available at: https://www.jisc.ac.uk/guides/designing-learning-and-assessment-in-a-digital-age
Karaman, S. (2011) ‘Examining the effects of flexible online exams on students’ engagement in e-learning’, Educational Research and Reviews, vol. 6, pp. 259–264.
Karim, N. A. & Shukur, Z. (2016) ‘Proposed features of an online examination interface design and its optimal values’, Computers in Human Behaviour, vol. 64, pp. 414–422. doi: 10.1016/j.chb.2016.07.013
Khare, A. & Lam, H. (2008) ‘Assessing student achievement and progress with online examinations: some pedagogical and technical issues’, International Journal on E-learning, vol. 7, pp. 383–402.
Marshall, S. (2010) ‘Change, technology and higher education: are universities capable of organisational change?’, ALT-J, Research in Learning Technology, vol. 18, pp. 179–192. doi: 10.1080/09687769.2010.529107
Milone, A. S., et al., (2017) ‘The impact of proctored online exams on the educational experience’, Currents in Pharmacy Teaching and Learning, vol. 9, pp. 108–114. doi: 10.1016/j.cptl.2016.08.037
Myyry, L. & Joutsenvirta, T. (2015) ‘Open-book, open-web online examinations: developing examination practices to support university students’ learning and self-efficacy’, Active Learning in Higher Education, vol. 16, pp. 119–132. doi: 10.1177/1469787415574053
Newhouse, C. P. (2011) ‘Using IT to assess IT: towards greater authenticity in summative performance assessment’, Computers and Education, vol. 56, pp. 388–402. doi: 10.1016/j.compedu.2010.08.023
O’Keeffe, C. (2016) ‘Producing data through e-assessment: a trace ethnographic investigation into e-assessment events’, European Educational Research Journal, vol. 15, pp. 99–116. doi: 10.1177/1474904115612486
Orlikowski, W. (2010) ‘The sociomateriality of organisational life: considering technology in management research’, Cambridge Journal of Economics, vol. 34, pp. 125–141. doi: 10.1093/cje/bep058
Orlikowski, W. J. & Scott, S. V. (2008) ‘Sociomateriality: challenging the separation of technology, work and organization’, Academy of Management Annals, vol. 2, no. 1, pp. 433–474. doi: 10.1080/19416520802211644
O’Shea, C. & Fawns, T. (2014) ‘Disruptions and dialogues: supporting collaborative connoisseurship in digital environments’, in Advances and Innovations in University Assessment and Feedback, eds C. Kreber et al., Edinburgh University Press, Edinburgh, pp. 259–273.
Ripley, M. (2009) ‘Transformational computer-based testing’, in The Transition to Computer-based Assessment, eds F. Scheuermann & J. Bjornsson, European Commission Joint Research Centre, Ispra, Italy, pp. 92–98.
Rust, C. (2007) ‘Towards a scholarship of assessment’, Assessment and Evaluation in Higher Education, vol. 32, pp. 229–237. doi: 10.1080/02602930600805192
Schmidt, S. M. P., Ralph, D. L. & Buskirk, B. (2009) ‘Utilizing online exams: a case study’, Journal of College Teaching and Learning, vol. 6, pp. 1–8. doi: 10.19030/tlc.v6i8.1108
Tracy, S. J. (2013) Qualitative Research Methods: Collecting Evidence, Crafting Analysis, Communicating Impact, Blackwell Publishing, Oxford.
Ullah, A., Ziao, H. & Barker, T. (2019) ‘A study into the usability and security implications of text and image based challenge questions in the context of online examination’, Education and Information Technologies, vol. 24, pp. 13–39. doi: 10.1007/s10639-018-9758-7
Williams, J. B. & Wong, A. (2009) ‘The efficacy of final exams: a comparative study of closed-book, invigilated exams and open-book, open-web exams’, British Journal of Educational Technology, vol. 40, pp. 227–236. doi: 10.1111/j.1467-8535.2008.00929.x
Footnotes
1 ‘Online exams’ are understood here as high-stakes summative assessment events, mediated by digital technologies, often taking place in a defined place or time and under secure conditions (e.g. invigilation, restrictions on access to course materials, notes or communication).
2 Myyry and Joutsenvirta (2015) acknowledged that ‘the specific context and tradition of university examinations in Finland’ (p. 129) might have influenced their results.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors contributing to Research in Learning Technology retain the copyright of their article and at the same time agree to publish their articles under the terms of the Creative Commons CC-BY 4.0 License (http://creativecommons.org/licenses/by/4.0/) allowing third parties to copy and redistribute the material in any medium or format, and to remix, transform, and build upon the material, for any purpose, even commercially, under the condition that appropriate credit is given, that a link to the license is provided, and that you indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
.png){kind=spacer}
