Examining educational technology and research impact: the two roles of e-learning and related terms in the 2014 REF impact case studies

Keywords: research impact, research assessment, educational technology, e-learning, learning technologists


The need to demonstrate the impact of research has become an important issue in the Higher Education sector in the UK. This has been taken care of through the introduction of ‘impact case studies’ as part of the research excellence framework (REF). The aim of the study presented in this paper was to understand the role that educational technology (and related terms) played in the 2014 REF impact case studies, using the public online database of case studies as a data source. Searches for 11 educational technology-related terms yielded a sample of 125 unique case studies. Although this represents only 1.9% of the total case studies, educational technology is clearly playing a role. The cases comprised two major subgroups: those where educational technology was the focus of the research (mainly associated with cases in education and computer science), and those where educational technology was used as a route to achieving impact (mainly in health-related subjects). The findings have implications for the contributions that educational technology and educational technologists can make in enhancing and supporting this important issue within their institutions.


Download data is not yet available.


Ackland, R. (2013) Web Social Science: Concepts, Data and Tools for Social Scientists in the Digital Age. London: Sage.

ALT. (2018) ‘What is learning technology?’, The Association for Learning Technology, [online] Available at: https://www.alt.ac.uk/about-alt/what-learning-technology

Bastian, M., Heymann, S. & Jacomy, M. (2009) ‘Gephi: an open source software for exploring and manipulating networks’, Proceedings of the International AAAI Conference on Weblogs and Social Media. San Jose, CA, USA, pp. 361–362.

Bendix, F., Kosara, R. & Hauser, H. (2006) ‘Parallel sets: interactive exploration and visual analysis of categorical data’, IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 4, pp. 558–568. doi: 10.1109/TVCG.2006.76

Biri, D., Oliver, K. & Cooper, A. (2014) ‘What is the impact of BEAMS research? An evaluation of REF impact case studies from UCL BEAMS. (STEaPP Working Paper)’, Department of Science, Technology, Engineering, and Public Policy, University College London, [online] Available at: http://discovery.ucl.ac.uk/1458641/

Cain, T. & Allan, D. (2017) ‘The invisible impact of educational research’, Oxford Review of Education, vol. 43, no. 6, pp. 718–732. doi: 10.1080/03054985.2017.1316252

Charmaz, K. (2014) Constructing Grounded Theory: A Practical Guide through Qualitative Analysis, 2nd ed. London: Sage.

Conole, G., et al., (2010) ‘Technology-enhanced learning as a site for interdisciplinary research’, Technology Enhanced Learning Research Programme: Institute of Education, London, [online] Available at: http://oro.open.ac.uk/35300/

Cotton, D. R. E., Miller, W. & Kneale, P. (2018) ‘The Cinderella of academia: is higher education pedagogic research undervalued in UK research assessment?’, Studies in Higher Education, vol. 43, no. 9, pp. 1625–1636. doi: 10.1080/03075079.2016.1276549

Digital Science. (2016) ‘Publication patterns in research underpinning impact in REF2014: a report to HEFCE by digital science: July 2016.’, Higher Education Funding Council for England (HEFCE), [online] Available at: http://dera.ioe.ac.uk/id/eprint/26933

Duncan, S. & Manners, P. (2017) ‘Engaging publics with research: reviewing the REF impact case studies and templates’, Bristol/NCCPE, [online] Available at: https://www.publicengagement.ac.uk/sites/default/files/publication/nccpe_ref_review_executive_summary.pdf

Else, H. (2017) ‘Impact weighting increased in UK’s REF 2021’, Times Higher Education, [online] Available at: https://www.timeshighereducation.com/news/impact-weighting-increased-uks-ref-2021

Glaser, B. G. & Strauss, A. L. (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, IL: Aldine.

Greenhalgh, T. & Fahy, N. (2015) ‘Research impact in the community-based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework’, BMC Medicine, vol. 13, no. 232. doi: 10.1186/s12916-015-0467-4

Havergal, C. (2019) ‘Australia releases first results of impact and engagement audit’, Times Higher Education [online] Available at: https://www.timeshighereducation.com/news/australia-releases-first-results-impact-and-engagement-audit

Hinrichs, S. & Grant, J. (2015) ‘A new resource for identifying and assessing the impacts of research’, BMC Medicine, vol. 13, no. 148. doi: 10.1186/s12916-015-0364-x

Jordan, K. & Carrigan, M. (2018) How was social media cited in 2014 REF Impact Case Studies? LSE Impact of Social Sciences Blog, [online] Available at: http://blogs.lse.ac.uk/impactofsocialsciences/2018/06/06/how-was-social-media-cited-in-2014-ref-impact-case-studies/.

Jump, P. (2013) ‘Evolution of the REF’, Times Higher Education, [online] Available at: https://www.timeshighereducation.com/features/evolution-of-the-ref/2008100.article

Kamenetzky, A., et al., (2016) ‘An analysis of the impact of research supported by the UK National Institute of Health Research’, Proceedings of Impacts of agricultural research - an approach of societal values: An International Conference organized by the French National Institute for Agricultural Research (INRA), [online], Available at: https://kclpure.kcl.ac.uk/portal/files/52658433/Impact_of_NIHR_research.pdf

Kelly, D., et al., (2016) ‘Impact case studies submitted to REF 2014: the hidden impact of nursing research’, Journal of Research in Nursing, vol. 21, no. 4, pp. 256–268. doi: 10.1177/1744987116646153

Khazragui, H. & Hudson, J. (2015) ‘Measuring the benefits of university research: impact and the REF in the UK’, Research Evaluation, vol. 24, no. 1, pp. 51–62. doi: 10.1093/reseval/rvu028

King’s College London and Digital Science (2015) ‘The nature, scale and beneficiaries of research impact: an initial analysis of Research Excellence Framework (REF) 2014 impact case studies’, HEFCE, [online] Available at: http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/Analysis,of,REF,impact/Analysis_of_REF_impact.pdf

Loach, T., Adams, J. & Szomszor, M. (2016) ‘Digital research report: the societal and economic impacts of academic research – International perspectives on good practice and managing evidence’, Digital Science, [online] Available at: 10.6084/m9.figshare.3117928.v2

Lupton, R. C. & Allwood, J. M. (2017) ‘Hybrid Sankey diagrams: visual analysis of multidimensional data for understanding resource use’, Resources, Conservation and Recycling, vol. 124, pp. 141–151. doi: 10.1016/j.resconrec.2017.05.002

Marcella, R., Lockerbie, H. & Bloice, L. (2016) ‘Beyond REF 2014: the impact of impact assessment on the future of information research’, Journal of Information Science, vol. 42, no. 3, pp. 369–385. doi: 10.1177/0165551516636291

Morse, J. (2007) ‘Theoretical saturation’, in The SAGE encyclopaedia of social science research methods, eds M. S. Lewis-Beck, A. Bryman & T. F. Liao, Sage, London, UK, p. 1123.

Ravenscroft, J., et al., (2017) ‘Measuring scientific impact beyond academia: an assessment of existing impact metrics and proposed improvements’, PLoS One, vol. 12, no. 3, e0173152. doi: 10.1371/journal.pone.0173152

Reed, S. & Kerridge, S. (2017) ‘How much was an impact case study worth in the UK Research Excellence Framework?’, Fast Track Impact blog, [online] Available at: http://www.fasttrackimpact.com/single-post/2017/02/01/How-much-was-an-impact-case-study-worth-in-the-UK-Research-Excellence-Framework

REF. (2014) ‘Impact case studies’, Research Excellence Framework 2014, [online] Available at: http://impact.ref.ac.uk/CaseStudies/

REF. (2018) ‘Draft guidance on submissions’, Research Excellence Framework website, [online] Available at: https://www.ref.ac.uk/media/1016/draft-guidance-on-submissions-ref-2018_1.pdf

Robbins, P. T., Wield, D. & Wilson, G. (2016) ‘Mapping engineering and development research excellence in the UK: an analysis of REF2014 impact case studies’, Journal of International Development, vol. 29, no. 1, pp. 89–105. doi: 10.1002/jid.3255

Ross, F. & Morrow, E. M. (2016) ‘Mining the REF impact case studies for lessons on leadership, governance and management in Higher Education’, LSE Impact of Social Sciences Blog, [online] Available at: http://blogs.lse.ac.uk/impactofsocialsciences/2016/06/08/leadership-governance-and-management-research-mining-the-ref-impact-case-studies/

Simpson, B. (2015) ‘REF 2014 and impact: reading the runes for anthropology in action’, Anthropology in Action, vol. 22, no. 2, pp. 1–4. doi: 10.3167/aia.2015.220201

Sivertson, G. (2017) ‘Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective’, Palgrave Communications, vol. 3, Article number: 17078. doi: 10.1057/palcomms.2017.78

Smith, K. E. & Stewart, E. (2017) ‘We need to talk about impact: why social policy academics need to engage with the UK’s research impact agenda’, Journal of Social Policy, vol. 46, no. 1, pp. 109–127. doi: 10.1017/S0047279416000283

Syed, J. & Davies, J. (2016) ‘Diversity in the authorship of journal articles and REF impact case studies: how are UK business schools shaping up?’ BAM2016 Conference Proceedings, [online], Available at: http://eprints.hud.ac.uk/id/eprint/29379/

Tusting, K. (2018) ‘The genre regime of research evaluation: contradictory systems of value around academics' writing’, Language and Education, vol. 32, no. 6, pp. 477–493. doi: 10.1080/09500782.2018.1505905

Tusting, K., et al., (2019) Academics Writing: The Dynamics of Knowledge Creation, Routledge, London.

UKRI. (2019) ‘How we fund research’, Research England website, [online] Available at: http://re.ukri.org/research/how-we-fund-research/

Vasant, S. (2014) ‘What is a learning technologist?’, Jobs.ac.uk website, [online] Available at: https://blog.jobs.ac.uk/education/teaching-learning/what-is-a-learning-technologist/

Watermeyer, R. (2012) ‘Issues in the articulation of ‘impact’: the responses of UK academics to ‘impact’ as a new measure of research assessment’, Studies in Higher Education, vol. 39, no. 2, pp. 359–377. doi: 10.1080/03075079.2012.709490

Watermeyer, R. & Lewis, J. (2017) ‘Institutionalizing public engagement through research in UK universities: perceptions, predictions and paradoxes concerning the state of the art’, Studies in Higher Education, vol. 43, no. 9, pp. 1612–1624. doi: 10.1080/03075079.2016.1272566

Wilkinson, C. (2019) ‘Evidencing impact: a case study of UK academic perspectives on evidencing research impact’,. Studies in Higher Education, vol. 44, no. 1, pp. 72–85. doi: 10.1080/03075079.2017.1339028

Woolridge, J. (2017) ‘Altmetrics linked to 3* and 4* impact scores in REF2014 impact case studies’, Altmetric blog, [online] Available at: https://www.altmetric.com/blog/altmetrics-ref2014-impact-case-studies/
How to Cite
Jordan, K. (2020). Examining educational technology and research impact: the two roles of e-learning and related terms in the 2014 REF impact case studies. Research in Learning Technology, 28. https://doi.org/10.25304/rlt.v28.2306
Original Research Articles