Playing the game: a realist approach to evaluating online student access, retention, progression and attainment initiatives
This study presents an evaluation of an online game-based student access, retention, progression and attainment (ARPA) initiative at the University of Kent. The initiative, a narrative-based simulation of a condensed student journey from pre-enrolment to graduation, is designed to prepare and support students in their transition to and participation in Higher Education. Student retention continues to be a perennial issue across the Higher Education sector, and studies have indicated that the more knowledgeable and informed students are about their university environment, the less likely they are to leave before completing their studies. Many institutions have developed interventions with the express purpose of addressing these concerns. Recognising the contextualised and subjective nature of such interventions, a realist evaluative framework was adopted to better understand the initiative under scrutiny, asking what works, for whom and in what circumstances. Participant interviews were utilised to assess the efficacy of the initiative in supporting students and in helping them to navigate often unfamiliar institutional cultures, practices and expectations. A revised programme theory is presented, enabling deeper insight into the merit of the initiative and its overall worth as a mechanism for change within the ARPA paradigm.
Advance HE. (2019) Essential Frameworks for Enhancing Student Success. York: Advance HE. Available at: https://www.advance-he.ac.uk/sites/default/files/2020-05/Enhancing%20Student%20Success%20in%20Higher%20Education%20Framework.pdf
Astbury, B. & Leeuw, F. L. (2010) ‘Unpacking black boxes: mechanisms and theory building in evaluation’, American Journal of Evaluation, vol. 31, no. 3, pp. 363–381. doi: 10.1177/1098214010371972
Billing, D. (1997) ‘Induction of new students to higher education’, Innovations in Education and Training International, vol. 34, no. 2, pp. 125–134. doi: 10.1080/1355800970340208
Bourdieu, P. (1977) Reproduction in Education, Society and Culture, SAGE Publications, London.
Brooman, S. & Darwent, S. (2014) ‘Measuring the beginning: a quantitative study of the transition to higher education’, Studies in Higher Education, vol. 39, no. 9, pp. 1523–1541. doi: 10.1080/03075079.2013.801428
Burke, P. J. (2012) The Right to Higher Education – Beyond Widening Participation, Routledge, London.
Buzzo, D., & Phelps, P. (2016) ‘JourneyMap: Visualising the time-bound student journey’, in EVA London 2016: Electronic Visualisation and the Arts (British Computer Society), eds N. Lambert, J. Bowen & G. Diprose, London, 12–15 July, pp. 178–185.
Carenys, J. & Moya, S. (2016) ‘Digital game-based learning in accounting and business education’, Accounting Education, vol. 25, no. 6, pp. 598–651. doi: 10.1080/09639284.2016.1241951
Cook, T. & Rushton, B. (2009) How to Recruit and Retain Higher Education Students – A Handbook of Good Practice, Taylor & Francis, London.
Crafter, S. & Maunder, R. (2012) ‘Understanding transitions using a sociocultural framework’, Educational and Child Psychology, vol. 29, no. 1, pp. 10–18.
Crosling, G., Heagney, M. & Thomas, L. (2009) ‘Improving student retention in higher education: improving teaching and learning’, The Australian Universities’ Review, vol. 51, no. 2, pp. 9–18. doi: 10.4324/9780203935453
Crozier, G. & Reay, D. (2008) ‘The socio cultural and learning experiences of working class students in higher education’, Teaching and Learning Research Briefing, (44) June. Teaching and Learning Research Programme, London.
Currant, B. & Keenan, C. (2009) ‘Evaluating systematic transition to higher education’, The Brookes Ejournal of Learning and Teaching, vol. 2, no. 4, pp. 1–12.
Dalkin, S., et al., (2021) ‘Using computer assisted qualitative data analysis software (CAQDAS; NVivo) to assist in the complex process of realist theory generation, refinement and testing’, International Journal of Social Research Methodology, vol. 24, no. 1, pp. 123–134. doi: 10.1080/13645579.2020.1803528
Dalkin, S. M., et al., (2012) ‘Understanding integrated care pathways in palliative care using realist evaluation: a mixed methods study protocol’, BMJ Open, vol. 2, no. 4. doi: 10.1136/bmjopen-2012-001533
Dalkin, S. M., et al., (2015) ‘What’s in a mechanism? Development of a key concept in realist evaluation’, Implementation Science, vol. 10, no. 1, p. 49. doi: 10.1186/s13012-015-0237-x
Davies, P. & Williams, J. (2001) ‘For me or not for me? Fragility and risk in mature students’ decision-making’, Higher Education Quarterly, vol. 55, no. 2, pp. 185–203. doi: 10.1111/1468-2273.00182
Dytham, S. & Hughes, C. (2017) Widening Participation Research And Evaluation: Where Are We Now?, University of Warwick, Coventry, UK, p. 8.
Emmel, N. (2013) Sampling and Choosing Cases in Qualitative Research: A Realist Approach, SAGE, London.
Formby, A., Woodhouse, A. & Basham, J. (2020) ‘Reframing widening participation towards the community: a realist evaluation’, Widening Participation and Lifelong Learning, vol. 22, no. 2, pp. 184–204. doi: 10.5456/WPLL.22.2.184
Francis, J. J., et al., (2010) ‘What is an adequate sample size? Operationalising data saturation for theory-based interview studies’, Psychology & Health, vol. 25, no. 10, pp. 1229–1245. doi: 10.1080/08870440903194015
Funnell, S. C. & Rogers, P. J. (2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models, Wiley, New York.
Guest, G., Bunce, A. & Johnson, L. (2006) ‘How many interviews are enough? an experiment with data saturation and variability’, Field Methods, vol. 2006, no. 18, pp. 59–82. doi: 10.1177/1525822X05279903
Hamshire, C., Whitton, N. & Whitton, P. (2012) ‘Staying the course - A game to facilitate students’ transitions to higher education’. In 6th European Conference on Games Based Learning, Cork, ed P. Felicia, Ireland, 4–5 October, pp. 624–630.
Harries, E., Hodgson, L. & Noble, J. (2014) Creating Your Theory of Change, NPC, London.
HEPI. (2021) A Short Guide to Non-Continuation in UK Universities, Higher Education Policy Institute, Oxford.
HESA. (2021) Non-Continuation: UK Performance Indicators | HESA. Available at: https://www.hesa.ac.uk/data-and-analysis/performance-indicators/non-continuation
Kapp, K. M. (2013) The Gamification of Learning and Instruction Fieldbook, John Wiley & Sons, New Jersey.
Keenan, C. (2009) ‘Stepping Stones 2HE: fresh thinking for introducing PDP to freshers’, in Enhancing Student-Centred Learning in Business and Management Hospitality Leisure Sport Tourism, eds J. Buswell & N. Becket, Threshold Press, Newbury, pp. 169–178.
Knowlton, L. W. & Phillips, C. C. (2012) The Logic Model Guidebook: Better Strategies for Great Results, SAGE, London.
Knox, H. (2005) ‘Making the transition from further to higher education: the impact of a preparatory module on retention, progression and performance’, Journal of Further and Higher Education, vol. 29, no. 2, pp. 103–110. doi: 10.1080/03098770500103135
Krause, M. & Williams, J. (2015) ‘A Playful Game Changer: Fostering Student Retention in Online Education with Social Gamification’. In Second ACM Conference on Learning @ Scale, Association for Computing Machinery, ed G. Kiczales, New York, 14–18 March, pp. 95–102. https://doi.org/10.1145/2724660.2724665
Laing, C., Robinson, A. & Johnston, V. (2005) ‘Managing the transition into higher education: an on-line spiral induction programme’, Active Learning in Higher Education, vol. 6, no. 3, pp. 243–255. doi: 10.1177/1469787405059575
Lefever, R. & Currant, B. (2010) ‘How can technology be used to improve the learner experience at points of transition’, Higher Education Academy, York, pp. 1–90
Lowe, H. & Cook, A. (2003) ‘Mind the gap: are students prepared for higher education?’, Journal of Further and Higher Education, vol. 27, no. 1, pp. 53–76. doi: 10.1080/03098770305629
Manzano, A. (2016) ‘The craft of interviewing in realist evaluation’, Evaluation, vol. 22, no. 3, pp. 342–360. doi: 10.1177/1356389016638615
Martin, C. & Benton, T. (2017) ‘Character creation: gamification and identity’, Teaching Media Quarterly, vol. 5, no. 2, pp. 1–8.
Money, J., et al., (2016) ‘Co-creating a blended learning curriculum in transition to higher education: a student viewpoint’, Creative Education, vol. 7, pp. 1205–1213. doi: 10.4236/ce.2016.79126
Morgan, M., et al., (2002) Risk Communication: A Mental Models Approach, Cambridge University Press, New York.
Moseley, A. (2008) ‘An alternate reality for higher education? Lessons to be learned from online games’. In ALT-C, ed S. Schmoller, Leeds, 9–11 September, pp. 92–95
O’Donnell, P., Kean, M. & Stevens, G. (2016) Student Transition in Higher Education – Concepts, Theories and Practices, Higher Education Academy, York.
Office for Students. (2018) Providing Information, Advice and Guidance for Students – Our Research. Available at: https://www.officeforstudents.org.uk/advice-and-guidance/student-information-and-data/providing-information-advice-and-guidance-for-students/our-research/
Office for Students. (2020) Information, Advice and Guidance for Prospective Students. Available at: https://www.officeforstudents.org.uk/publications/coronavirus-briefing-note-information-advice-and-guidance-for-prospective-students/
Pawson, R. (2013) The Science of Evaluation – A Realist Manifesto, SAGE Publications, London.
Pawson, R. & Tilley, N. (1997) Realistic Evaluation, SAGE Publications, London.
Pennington, C. R., et al., (2018) ‘Transitioning in higher education: an exploration of psychological and contextual factors affecting student satisfaction’, Journal of Further and Higher Education, vol. 42, no. 5, pp. 596–607. doi: 10.1080/0309877X.2017.1302563
Piatt, K. (2009) ‘Using alternate reality games to support first year induction with ELGG’, Campus-Wide Information Systems, vol. 26, no. 4, pp. 313–322. doi: 10.1108/10650740910984646
Pickering, N. (2021) ‘Enabling equality of access in higher education for underrepresented groups: a realist “small step” approach to evaluating widening participation’, Research in Post-Compulsory Education, vol. 26, no. 1, pp. 111–130. doi: 10.1080/13596748.2021.1873410
Reay, D., David, M. E. & Ball, S. (2005) Degrees of Choice: Social Class, Race and Gender in Higher Education, Trentham Books Ltd, Stoke on Trent, UK.
Robbins, D. (1993) ‘The practical importance of Bourdieu’s analyses of Higher Education’, Studies in Higher Education, vol. 18, no. 2, pp. 151–163. doi: 10.1080/03075079312331382339
Roberts, C., et al., (2003) ‘Supporting student ‘success’: what can we learn from the persisters?’, in Education in a Changing Environment: Inaugural Learning and Teaching Research Conference, eds L. Anderson, 17–18 September, University of Salford, UK, pp. 45–53.
Ryan, F. (2020) ‘A virtual law clinic: a realist evaluation of what works for whom, why, how and in what circumstances?’, The Law Teacher, vol. 54, no. 2, pp. 237–248. doi: 10.1080/03069400.2019.1651550
Sailer, M., et al., (2017) ‘How gamification motivates: an experimental study of the effects of specific game design elements on psychological need satisfaction’, Computers in Human Behavior, vol. 69, pp. 371–380. doi: 10.1016/j.chb.2016.12.033
Tchen, P., et al., (2018) ‘Bridging the gap: an evaluation of self-paced online transition modules for advanced pharmacy practice experience students’, Currents in Pharmacy Teaching and Learning, vol. 10, no. 10, pp. 1375–1383. doi: 10.1016/j.cptl.2018.07.006
Turner, R., et al., (2017) ‘Easing the transition of first year undergraduates through an immersive induction module’, Teaching in Higher Education, vol. 22, no. 7, pp. 805–821. doi: 10.1080/13562517.2017.1301906
University of Kent. (2019a) ‘One Hour Degree proposal’. Internal University of Kent paper. Unpublished.
University of Kent. (2019b) Student Success Project – The One Hour Degree. Available at: https://www.kent.ac.uk/studentsuccess/onehourdegree.html
Webb, O. J. & Cotton, D. R. E. (2018) ‘Early withdrawal from higher education: a focus on academic experiences’, Teaching in Higher Education, vol. 23, no. 7, pp. 835–852. doi: 10.1080/13562517.2018.1437130
Weiss, C. (1997) ‘Theory-based evaluation: past, present, and future’, New Directions for Evaluation, vol. 1997, no. 76, pp. 41–55. doi: 10.1002/ev.1086
Whitton, N., et al., (2014) ‘Alternate reality games as learning environments for student induction’, Interactive Learning Environments, vol. 22, no. 3, pp. 243–252. doi: 10.1080/10494820.2011.641683
Yorke, M. & Longden, B. (2008) The First-Year Experience of Higher Education in the UK – Final Report, Higher Education Academy, York.
Zahedi, L., et al., (2021) ‘Gamification in education: a mixed-methods study of gender on computer science students’ academic performance and identity development’, Journal of Computing in Higher Educuation, 33, pp. 441–474. doi: 10.1007/s12528-021-09271-5
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors contributing to Research in Learning Technology retain the copyright of their article and at the same time agree to publish their articles under the terms of the Creative Commons CC-BY 4.0 License (http://creativecommons.org/licenses/by/4.0/) allowing third parties to copy and redistribute the material in any medium or format, and to remix, transform, and build upon the material, for any purpose, even commercially, under the condition that appropriate credit is given, that a link to the license is provided, and that you indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.