ORIGINAL RESEARCH ARTICLE
Examining educational technology and research impact: the two roles of e-learning and related terms in the 2014 REF impact case studies
Katy Jordan*
Faculty of Education, University of Cambridge, UK
(Received: 23 July 2019; Revised: 31 December 2019; Accepted: 10 February 2020; Published: 10 March 2020)
Abstract
The need to demonstrate the impact of research has become an important issue in the Higher Education sector in the UK. This has been taken care of through the introduction of ‘impact case studies’ as part of the research excellence framework (REF). The aim of the study presented in this paper was to understand the role that educational technology (and related terms) played in the 2014 REF impact case studies, using the public online database of case studies as a data source. Searches for 11 educational technology-related terms yielded a sample of 125 unique case studies. Although this represents only 1.9% of the total case studies, educational technology is clearly playing a role. The cases comprised two major subgroups: those where educational technology was the focus of the research (mainly associated with cases in education and computer science), and those where educational technology was used as a route to achieving impact (mainly in health-related subjects). The findings have implications for the contributions that educational technology and educational technologists can make in enhancing and supporting this important issue within their institutions.
Keywords: research impact; research assessment; educational technology; e-learning; learning technologists
*Corresponding author. Email: klj33@cam.ac.uk
Research in Learning Technology 2020. © 2020 K. Jordan. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.
Citation: Research in Learning Technology 2020, 28: 2306 - http://dx.doi.org/10.25304/rlt.v28.2306
Introduction
The issue of ‘research impact’ is of critical importance to the Higher Education sector in the UK and is dealt with by using the research excellence framework (REF). The REF is a national audit of research outputs, and its result has implications for the levels of funding that institutions will receive. The REF was launched in 2014, replacing a previous model known as the research assessment exercise (RAE). One of the key distinctions between the REF and the RAE was the foregrounding of the perceived ‘real world’ impact of research, framed as a reflection of better ‘value’ for research funding (Jump 2013). As a result, the 2014 REF was the first assessment exercise to include submissions of ‘impact case studies’ as a mechanism for representing and assessing evidence of impact. Whilst such impacts may be challenging to define and it is debatable whether the case study format accurately captures impact (Khazragui and Hudson 2015), academics do perceive its inclusion to be valuable in presenting a richer account of scholarly activity (Watermeyer 2012).
In the 2014 REF, 6975 case studies were submitted, 6679 of which were subsequently published online in a database (REF 2014). The database represents a valuable resource for further research and analysis in relation to research impact. As 2014 was the year of the introduction of REF, the sector is actively learning and adapting to this new form of output (Tusting et al. 2019), and the database has been a source of information for several cross-cutting analyses to highlight the role of impact case studies in practice.
The importance of impact case studies has been underlined by quantifying the link to funding received by institutions. Although the database does not include the results of the assessment for each case study, information may be inferred from the average scores of units of assessment. Reed and Kerridge (2017) examine the link between units of assessment which obtained a score of 3* or 4* (the highest grades conferred by the REF) for their impact case studies in the 2014 REF and their subsequent QR funding allocation in 2016/2017, concluding that on average a 4* case study was worth approximately £35000 more than a 3* one in funding terms.
Trends in the database have also been examined in order to characterise what constitutes an impact case study. King’s College London and Digital Science (2015) used text-mining to consider over-arching trends in the database, coupled with addressing specific questions identified as being of key importance through workshops. The report notes that impact case studies are often multi-disciplinary in nature, and that different institutions appear to specialise in different types of impact to a certain extent (ibid). Further characterisation of trends in the database overall has interrogated the nature of impact within the case study format by examining the relationships between the cases, public engagement and traditional scholarly metrics. Public engagement has been shown to be frequently linked to impact case studies, but there is variation in the prevalence and form of public engagement across different disciplines (Duncan and Manners 2017; Watermeyer and Lewis 2017). Contrasting findings have been reported about the relationship between impact case studies and scholarly bibliometrics. With the addition of data about the scoring received by case studies in the REF, Woolridge (2017) demonstrates a link between highly rated case studies and altmetrics scores, while Ravenscroft et al. argue that the link between impact case studies and traditional citation-based metrics is weak (Ravenscroft et al. 2017). Social media also plays a role, which has been mentioned in approximately one quarter of the case studies (Jordan and Carrigan 2018).
While case studies are often multi-disciplinary (Digital Science 2016), the nature of impact varies according to discipline and subject area. In examining the types of evidence submitted in support of impact case studies, Loach, Adams and Szomszor (2016) report testimonials, reports and articles as being the most frequently used, and some disciplinary preferences emerged. For example, testimonials were the most frequently used evidence type, and used to a significantly greater extent in arts and humanities, while under-used overall in biological sciences and medicine (ibid). The converse is true of reports and articles, which are the second and third most frequently used evidence sources (ibid). Subjects also vary in the extent to which the papers submitted as supporting evidence in case studies are also submitted to the main body of the REF, with more applied ‘units of assessment’ demonstrating greater overlap (Digital Science 2016). Several studies have examined REF impact case studies in specific subjects, for example, from the perspectives of anthropology (Simpson 2015), business (Syed and Davies 2016), educational research (Cain and Allan 2017; Cotton, Kneale, and Miller 2018), engineering (Biri, Oliver, and Cooper 2014; Robbins, Wield, and Wilson 2016), health (Greenhalgh and Fahy 2015; Hinrichs and Grant 2015; Kamenetzky et al. 2016; Kelly et al. 2016), leadership, governance and management (Ross and Morrow 2016), library and information science (Marcella, Lockerbie, and Bloice 2016) and social work (Smith and Stewart 2017). As the 2014 REF was the first to make use of the impact case studies format, different subject areas have sought to learn lessons from the database about the character of impact case studies in their subjects to reflect on the nature of impact and prepare for the next REF, when the weighting of case studies will be increased from 20% to 25% (Else 2017).
There is a gap in the current literature in relation to educational technology and research impact. Examining the role that educational technology and related terms played in the 2014 REF, impact case studies can provide useful insights and practical implications for the field in two main ways. Firstly, they help in understanding how research into educational technology and related topics were located within the 2014 REF. This is a particularly important question as technology-enhanced learning is an interdisciplinary field (Conole et al. 2010). Secondly, they provide insights into the ways in which educational technologists can support academics and practitioners from across the disciplines in relation to developing their REF submissions. The following questions guided the analysis presented here:
- How prevalent were educational technology-related terms in the 2014 REF impact case studies?
- Which types of case studies are educational technology-related terms associated with, in terms of subject areas and types of impact?
- What role do educational technology-related terms play in relation to impact case studies?
Methods
Data were sourced from the 2014 REF impact case studies database, which is a freely available online database (REF 2014). The database comprises 6679 non-redacted case studies, which represents 96% of the total 6975 case studies submitted to the 2014 REF (ibid). Database searches were conducted using the following search terms (Table 1). While there are many more search terms related to specific educational technologies or pedagogic innovations that could have been included, the terms were selected to ensure that the main synonyms for educational technology were covered. This strategy was adopted in order to maintain a focus on the role of educational technology in general.:
Information about the case studies returned by each search was exported as a series of spreadsheets. The spreadsheets were combined, duplicates removed and each case tagged with the search terms associated with it. Two cases were later removed due to false positive searches. Both cases were associated with the ‘learning technology’ search term, and both had been found due to the term ‘machine learning technology’ being present in the case study. The resulting sample comprised 125 case studies. Since the REF 2014 impact case studies database is a publicly available online resource, no ethical issues were associated with the work. However, although the cases do include names and other personally identifiable data including institutional affiliation, it would not add anything to the research questions at hand to include these data types here.
Once the sample had been constructed and prepared, each case was examined in turn by the researcher to identify where the corresponding search term was found, and to categorise the ways in which educational technology and the related terms were used. As the study was exploratory in nature, an open coding approach to categorisation was used (in the manner of grounded theory, but short of a full grounded theory-based design; Charmaz 2014; Glaser and Strauss 1967), rather than imposing a pre-existing coding scheme. The possibilities of basing a coding scheme on an existing typology of educational technologies or types of pedagogical context or approach were considered; however, neither were used, for two reasons. First, there was considerable variation in the level of detail available in the case studies, which would have posed a challenge for applying pre-existing typologies. Second, neither potential typology would have fully encompassed the focus of the research question here (the coding being used particularly to address research question 3), which is concerned with the role that the terms are playing in this context, for which there is no existing coding scheme. On this basis, open coding was used, to support the emergence of categories from the sample.
In coding the cases, two major distinct groups within the dataset quickly became apparent, and a sense of theoretical saturation emerged after approximately 25 cases (Morse 2007). Ultimately, analysis of the coded dataset was descriptive in nature, to examine trends in relation to subject area, types of impact and how educational technology-related terms are being used in this context. Links between categories within subject areas, impact types and the use of terms are presented in the analysis in a form derived from Sankey diagrams. While Sankey diagrams are a tool intended to represent the flow of resources within complex systems, typically manufacturing, the format has been co-opted by the field of data visualisation and can be applied to sets of categorical data more broadly (Bendix, Kosara, and Hauser 2006; Lupton and Allwood 2017).
Results
Research question 1
The first research question considered ‘How prevalent were educational technology-related terms in the 2014 REF impact case studies?’. The sample included 125 unique case studies, which represents 1.9% of the total 6679 cases in the database. While this may suggest that educational technology and related terms are relatively niche within this context, 125 is not an insubstantial number.
The number of case studies returned per search term are shown in Table 1. The distribution shows that the sample is dominated by use of the term ‘e-learning’, which alone accounts for approximately half of the case studies. It should be noted that the figures reflect the numbers per query, prior to de-duplication. A total of 34 of the cases (27.2%) contained two or more of the search terms; 11 (8.8%) contained between three and five. Visualising the information as a network graph is a useful way to explore patterns within those containing multiple search terms, as it allows multiple relationships between terms to be mapped. The co-occurrence of search terms within the de-duplicated sample is shown as a weighted network (Ackland 2013) rendered using Gephi (Bastian, Heymann, and Jacomy 2009) in Figure 1. In a weighted network, the links (‘edges’) between terms are assigned values based on the strength of the connection (Ackland 2013). The numbers of cases that included the pairs of search terms – and thus reflect the thickness of the edges in Figure 1 – are shown in Table 2. E-learning emerges as the most prominent term, most frequently associated with ‘elearning’, ‘educational technology’ and ‘technology-enhanced learning’. These characteristics suggest that there may be two distinct sub-groups within the sample; a majority which simply refer to ‘e-learning’, and a minority which use multiple ways of discussing educational technology.
| Edge | Weight |
| e-learning, elearning | 11 |
| e-learning, technology-enhanced learning | 11 |
| e-learning, educational technology | 10 |
| e-learning, learning technology | 5 |
| elearning, educational technology | 4 |
| educational technology, technology-enhanced learning | 4 |
| Educational technology, learning technology | 3 |
| Technology-enhanced learning, learning technology | 3 |
| e-learning, mobile learning | 3 |
| Mobile learning, technology-enhanced learning | 3 |
| Educational technology, elearning | 2 |
| e-learning, massive open online courses | 2 |
Figure 1. A network graph illustrating the co-occurrence of educational technology-related terms in the sample. Edges are weighted to reflect the number of cases which contain a pair of terms; node sizes reflect the number of cases which include a particular term. Edge weights are shown in Table 2.
Research question 2
The second question was concerned with examining the types of case studies in which educational technology-related terms are found. The database readily provides information about the type of impact, and subject area (‘unit of assessment’). While information about the submitting institution is also available, it was not included in the analysis because the number of case studies submitted varies by institution size, so trends may be misleading.
The number of cases included in the sample according to type of impact are shown in Table 3. As the total number of cases submitted illustrating different types of impact varies substantially (from 212 demonstrating ‘legal’ impact to 1723 demonstrating ‘societal’ impact), the figures are also given as a percentage of the total number in the database per impact type. Note that the typology of impact types is defined by the REF impact case studies database. Each case study within the database states the associated impact type in its introductory text (REF 2014). The cases containing educational technology-related terms are mainly associated with two types of impact: ‘societal’ and ‘health’.
Table 4 shows the representation of different subject areas (or ‘units of assessment’, to use the categorisation scheme adopted by the REF) within the dataset. For the REF, units of assessment are arranged into four broader disciplinary groups, or ‘panels’. The sample includes cases drawn from all four panels, to different extents. Panel B, which focuses upon the natural and physical Sciences, is least well-represented. Within the other panels, there are substantial differences according to unit of assessment. Three units stand out as having greater than 5% of their cases mentioning educational technology-related terms. These are: allied health professions, dentistry, nursing and pharmacy; computer science and informatics and education.
Research question 3
In order to address the third question, ‘what role do educational technology-related terms play in relation to impact case studies?’, the text of each case study was examined to identify and categorise how the search term was being used. Two distinct, over-arching categories emerged from the data: those in which educational technology was an explicit part of the focus of the underpinning research in the case study (57), and those in which educational technology had been used to facilitate dissemination of other research findings (65). Additionally, three case studies included the term ‘e-learning’ as part of a testimonial contributors’ job description. The two categories are clearly associated with a contrasting range of impact types (Figure 2). Furthermore, the categories are also associated primarily with different panels, with those where educational technology is used only for dissemination being mainly associated with panel A (Figure 2).
Figure 2. Relationships between the two main categories of impact case studies identified within the sample (shown in the middle), the type of impact presented by the cases (left) and the panels the cases fall within (right). Note that the three cases where ‘e-learning’ was only included as a testimonial affiliation are not shown.
The group of cases where educational technology was simply used as a route to achieving impact (referred to as ‘outputs’ for brevity in Figure 2) was quite homogenous, typically including a statement that an e-learning module or materials had been made available to disseminate the findings of the research. The type of impacts documented by the cases in this group included cultural (7), economic (2), environmental (2), health (24), legal (1), political (5), societal (19) and technological (5). The profile of the cases in this group according to unit of assessment is shown in Table 5. All but five of the 65 cases in this group were included on the basis of the keywords ‘e-learning’ or ‘elearning’. As such, this group of cases were not subject to further analysis.
For the 57 cases where educational technology was the focus of the research underpinning the case study itself (referred to as ‘research context’ in Figures 2 and 3), the type of impact being reported in this group was predominantly societal (46), and also cultural (5), health (1) and technological (5). The distribution of the 57 cases according to unit of assessment is shown in Table 6. Note that in comparison to Table 5, there is less variation in subject areas here, as the cases within this group are more frequently found within either the computer science or education panels.
Figure 3. Relationships between the four categories of impact case studies identified which focus on educational technology (shown in the middle), the type of impact presented by the cases (left) and the panels the cases fall within (right).
A second round of finer-grained classification was undertaken on the 57 cases in which educational technology was part of the focus of the research itself. As the scope of research being reported in case studies can vary from an individuals’ career, to a particular project, group or programme of research, this subsequent categorisation proved more challenging. The cases required two rounds of categorisation but nonetheless could be broadly divided into three categories based mainly on the positioning of educational technology in relation to the research context at hand. The following categories were identified, with the number of case studies contributing to each shown in brackets:
- Development and evaluation of subject-specific educational technology initiatives (21; 36.8%): This was the largest group within the sub-sample, and represents a more applied group of research settings. In this group, the role of educational technology was more elaborate than simply dissemination (as in the other sub-sample) but focused on deploying educational technologies in a range of different subject settings.
- Educational technology-focused education research (17; 29.8%): This was the group which would best align with research squarely defined as technology-enhanced learning research, applying rigorous educational research approaches to educational technologies or novel pedagogies associated with them. A range of settings was identified here, including serious games, mobile learning and open education, for example.
- Policy and development (5; 8.8%): This included a small but distinct group of cases where the research included reference to educational technology in the context of educational policy, or informing policy for educational development initiatives.
- Technological and app development (14; 24.6%): Cases where the focus is principally upon the development of technologies or apps that are related to educational technology, such as developing collaborative video technologies, technical specifications for e-learning or apps for citizen science.
As noted, the range of research covered by an impact case study can be broad. Cases were assigned to the category that best described their positioning in relation to educational technology and related terms, but a case study may have had links to several categories. The four sub-categories of case studies are also associated with different profiles in relation to the impact types reported, and submission panel (Figure 3). The number of case studies per sub-category and panel are shown in further detail according to unit of assessment in Table 7. Subject-specific educational technology initiatives span a wider range of subject areas and impact types. The categories of educational technology-focused educational research and technological and app development are both broadly split between panels C (education unit of assessment) and B (computer science unit of assessment). However, a key distinction is that while educational technology-focused educational research is associated exclusively with societal impact, technological and app development also includes technological and cultural impacts.
Discussion and conclusions
Across the levels of analysis presented here, educational technology and related terms played two distinct roles in the 2014 REF impact case studies. The sample of case studies from the REF impact case studies was approximately equally split between the two roles. On the one hand, educational technology-related research is being showcased through submission of REF impact case studies. These cases are associated with a wider range of educational technology-related terms, where the impact type is mainly ‘societal’ and principally found in cases assigned to the education unit of assessment (and to a lesser extent, computer science and informatics). On the other hand, educational technology plays a role in cases as a route to achieving impact. This is often framed in terms of ‘e-learning’ provision and associated with cases across a range of units of assessment, but notably to a greater extent in health professions and medicine.
Both characterisations have practical implications for learning technologists and their work within higher education institutions. Definitions of the term ‘learning technologist’ encompass a wide range of activities mediating between technologies, staff and students (ALT 2018; Vasant 2014). The influence of the REF has led to impact case studies becoming distinct emergent types of academic literacies (Tusting 2018; Tusting et al. 2019; Wilkinson 2019). If, as the analysis suggests, educational technology is being incorporated into this type of discourse, this is an area where learning technologists may require further training. For example, impact in the context of the REF is assessed in terms of both ‘reach’ and ‘significance’ of changes to practice (REF 2018). Simply making an e-learning module available through an online network with a known number of users may be an effective way of achieving reach, but this is only half of the picture. However, simple mechanisms for capturing significance (such as user surveys) could easily be incorporated at the design stage.
In terms of the potential for educational technology to enhance impact of research in general, the results suggest that educational technology may be under-used in relation to the REF and research impact at present, and that learning technologists could play an important role in further supporting this within institutions. Educational technology as a route to achieving impact was predominantly used in health and medicine-related subjects, as this is an area for which e-learning is already embedded for ongoing training and continuous professional development. Although the infrastructure and practices around e-learning are already established in this field, developing e-learning modules could be deployed as an impact strategy in any field, particularly through the development of open educational resources (OER) and massive open online courses (MOOCs). MOOCs in particular have seen enormous expansion since the 2014 REF assessment period and are likely to feature more extensively in the next iteration scheduled for 2021. It is not clear, however, how well the major MOOC platforms are designed to collect the information that would be most valuable to support a REF impact case study.
The findings also shed light upon the role of educational technology as a research field in itself. As the REF is a high-profile institutional exercise but has only been carried out once to date, the field is still in a nascent stage and adapting to the new format of impact case studies, and it is potentially useful to explore and reflect upon where educational technology-related research sits within this. Educational technology in relation to research impact in this context spans a range of roles between being the subject of fundamental research itself, to purely applied instances where it is included as a tool to enact research impact through. The alignment of educational technology-related research primarily with the education unit of assessment means that issues identified in relation to education may also be relevant. Studies of education impact case studies caution that for educational research initiatives to be valid examples of impact as defined by the REF, impacts within the same institution are not permissible and must be demonstrable in other institutions or contexts (Cain and Allan 2017; Cotton et al. 2018). As cases that report educational technology-related educational research are divided between the Education and Computer Science units of assessment, further analysis of the computer science cases would be useful in order to contrast with the existing work in education to see if there are implications associated with which unit cases are submitting to.
REF impact case studies and their associated literature are an area in which the role of professional networks of learning technologists, such as those facilitated by the Association of Learning Technology, could be particularly valuable as a way of supporting practitioners and promoting knowledge exchange between institutions. Given the competitive nature of the REF and its direct link to funding (UKRI 2019), institutions may be reluctant to share best practices, but would likely appreciate the opportunities for staff development, so this is an area where professional networks could provide support. Additionally, the cyclical nature of REF assessment periods mean that preparations intensify in the run-up to submissions. Personnel and expertise from one round may be lost in the years immediately following one of the exercises.
Limitations
This study is limited to an extent in that it is exclusively focused upon the UK. While REF impact case studies are a high-profile issue within the Higher Education sector in the UK, the model has not yet seen uptake in other countries (Sivertson 2017). One exception can be found in the Australian Higher Education sector, which has recently seen the introduction of the engagement and impact assessment, in addition to its main RAE (Havergal 2019). However, increasing the impact of research beyond academia is a desirable outcome whether it is formally audited or not, and a key issue for the learning technology community to engage with.
Conclusions
This exploratory analysis has shown that educational technology and related terms are feeding into REF impact case studies. While the number of cases is small when seen as a percentage of the whole database (1.9%), a substantial number (125) do contain educational technology-related terms. The cases are divided between instances where e-learning has been deployed as a route to achieving impact, and those where educational technology is the focus of the research. The use of educational technology as a tool to promote impact (by developing e-learning modules to communicate the findings of a research project, for example) is principally associated with subjects within the biological and medical sciences. The use of educational technology-based tools as a means of dissemination and enhancing impact may be useful for other disciplines and subject areas to adopt. However, simply developing tools is not sufficient to demonstrate impact in terms of the REF. Development for learning technologists may be required, to raise awareness of how e-learning tools could be designed in order to capture significance and reach. Educational technology-focused case studies are most frequently associated with education and computer science units of assessment and classed as demonstrating ‘societal’ impact. Further detailed analysis of this group of cases would be valuable in order to gain an in-depth understanding of how impact is achieved and demonstrated in this context.
References
Ackland, R. (2013) Web Social Science: Concepts, Data and Tools for Social Scientists in the Digital Age. London: Sage.
ALT. (2018) ‘What is learning technology?’, The Association for Learning Technology, [online] Available at: https://www.alt.ac.uk/about-alt/what-learning-technology
Bastian, M., Heymann, S. & Jacomy, M. (2009) ‘Gephi: an open source software for exploring and manipulating networks’, Proceedings of the International AAAI Conference on Weblogs and Social Media. San Jose, CA, USA, pp. 361–362.
Bendix, F., Kosara, R. & Hauser, H. (2006) ‘Parallel sets: interactive exploration and visual analysis of categorical data’, IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 4, pp. 558–568. doi: 10.1109/TVCG.2006.76
Biri, D., Oliver, K. & Cooper, A. (2014) ‘What is the impact of BEAMS research? An evaluation of REF impact case studies from UCL BEAMS. (STEaPP Working Paper)’, Department of Science, Technology, Engineering, and Public Policy, University College London, [online] Available at: http://discovery.ucl.ac.uk/1458641/
Cain, T. & Allan, D. (2017) ‘The invisible impact of educational research’, Oxford Review of Education, vol. 43, no. 6, pp. 718–732. doi: 10.1080/03054985.2017.1316252
Charmaz, K. (2014) Constructing Grounded Theory: A Practical Guide through Qualitative Analysis, 2nd ed. London: Sage.
Conole, G., et al., (2010) ‘Technology-enhanced learning as a site for interdisciplinary research’, Technology Enhanced Learning Research Programme: Institute of Education, London, [online] Available at: http://oro.open.ac.uk/35300/
Cotton, D. R. E., Miller, W. & Kneale, P. (2018) ‘The Cinderella of academia: is higher education pedagogic research undervalued in UK research assessment?’, Studies in Higher Education, vol. 43, no. 9, pp. 1625–1636. doi: 10.1080/03075079.2016.1276549
Digital Science. (2016) ‘Publication patterns in research underpinning impact in REF2014: a report to HEFCE by digital science: July 2016.’, Higher Education Funding Council for England (HEFCE), [online] Available at: http://dera.ioe.ac.uk/id/eprint/26933
Duncan, S. & Manners, P. (2017) ‘Engaging publics with research: reviewing the REF impact case studies and templates’, Bristol/NCCPE, [online] Available at: https://www.publicengagement.ac.uk/sites/default/files/publication/nccpe_ref_review_executive_summary.pdf
Else, H. (2017) ‘Impact weighting increased in UK’s REF 2021’, Times Higher Education, [online] Available at: https://www.timeshighereducation.com/news/impact-weighting-increased-uks-ref-2021
Glaser, B. G. & Strauss, A. L. (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, IL: Aldine.
Greenhalgh, T. & Fahy, N. (2015) ‘Research impact in the community-based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework’, BMC Medicine, vol. 13, no. 232. doi: 10.1186/s12916-015-0467-4
Havergal, C. (2019) ‘Australia releases first results of impact and engagement audit’, Times Higher Education [online] Available at: https://www.timeshighereducation.com/news/australia-releases-first-results-impact-and-engagement-audit
Hinrichs, S. & Grant, J. (2015) ‘A new resource for identifying and assessing the impacts of research’, BMC Medicine, vol. 13, no. 148. doi: 10.1186/s12916-015-0364-x
Jordan, K. & Carrigan, M. (2018) How was social media cited in 2014 REF Impact Case Studies? LSE Impact of Social Sciences Blog, [online] Available at: http://blogs.lse.ac.uk/impactofsocialsciences/2018/06/06/how-was-social-media-cited-in-2014-ref-impact-case-studies/.
Jump, P. (2013) ‘Evolution of the REF’, Times Higher Education, [online] Available at: https://www.timeshighereducation.com/features/evolution-of-the-ref/2008100.article
Kamenetzky, A., et al., (2016) ‘An analysis of the impact of research supported by the UK National Institute of Health Research’, Proceedings of Impacts of agricultural research - an approach of societal values: An International Conference organized by the French National Institute for Agricultural Research (INRA), [online], Available at: https://kclpure.kcl.ac.uk/portal/files/52658433/Impact_of_NIHR_research.pdf
Kelly, D., et al., (2016) ‘Impact case studies submitted to REF 2014: the hidden impact of nursing research’, Journal of Research in Nursing, vol. 21, no. 4, pp. 256–268. doi: 10.1177/1744987116646153
Khazragui, H. & Hudson, J. (2015) ‘Measuring the benefits of university research: impact and the REF in the UK’, Research Evaluation, vol. 24, no. 1, pp. 51–62. doi: 10.1093/reseval/rvu028
King’s College London and Digital Science (2015) ‘The nature, scale and beneficiaries of research impact: an initial analysis of Research Excellence Framework (REF) 2014 impact case studies’, HEFCE, [online] Available at: http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/Analysis,of,REF,impact/Analysis_of_REF_impact.pdf
Loach, T., Adams, J. & Szomszor, M. (2016) ‘Digital research report: the societal and economic impacts of academic research – International perspectives on good practice and managing evidence’, Digital Science, [online] Available at: 10.6084/m9.figshare.3117928.v2
Lupton, R. C. & Allwood, J. M. (2017) ‘Hybrid Sankey diagrams: visual analysis of multidimensional data for understanding resource use’, Resources, Conservation and Recycling, vol. 124, pp. 141–151. doi: 10.1016/j.resconrec.2017.05.002
Marcella, R., Lockerbie, H. & Bloice, L. (2016) ‘Beyond REF 2014: the impact of impact assessment on the future of information research’, Journal of Information Science, vol. 42, no. 3, pp. 369–385. doi: 10.1177/0165551516636291
Morse, J. (2007) ‘Theoretical saturation’, in The SAGE encyclopaedia of social science research methods, eds M. S. Lewis-Beck, A. Bryman & T. F. Liao, Sage, London, UK, p. 1123.
Ravenscroft, J., et al., (2017) ‘Measuring scientific impact beyond academia: an assessment of existing impact metrics and proposed improvements’, PLoS One, vol. 12, no. 3, e0173152. doi: 10.1371/journal.pone.0173152
Reed, S. & Kerridge, S. (2017) ‘How much was an impact case study worth in the UK Research Excellence Framework?’, Fast Track Impact blog, [online] Available at: http://www.fasttrackimpact.com/single-post/2017/02/01/How-much-was-an-impact-case-study-worth-in-the-UK-Research-Excellence-Framework
REF. (2014) ‘Impact case studies’, Research Excellence Framework 2014, [online] Available at: http://impact.ref.ac.uk/CaseStudies/
REF. (2018) ‘Draft guidance on submissions’, Research Excellence Framework website, [online] Available at: https://www.ref.ac.uk/media/1016/draft-guidance-on-submissions-ref-2018_1.pdf
Robbins, P. T., Wield, D. & Wilson, G. (2016) ‘Mapping engineering and development research excellence in the UK: an analysis of REF2014 impact case studies’, Journal of International Development, vol. 29, no. 1, pp. 89–105. doi: 10.1002/jid.3255
Ross, F. & Morrow, E. M. (2016) ‘Mining the REF impact case studies for lessons on leadership, governance and management in Higher Education’, LSE Impact of Social Sciences Blog, [online] Available at: http://blogs.lse.ac.uk/impactofsocialsciences/2016/06/08/leadership-governance-and-management-research-mining-the-ref-impact-case-studies/
Simpson, B. (2015) ‘REF 2014 and impact: reading the runes for anthropology in action’, Anthropology in Action, vol. 22, no. 2, pp. 1–4. doi: 10.3167/aia.2015.220201
Sivertson, G. (2017) ‘Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective’, Palgrave Communications, vol. 3, Article number: 17078. doi: 10.1057/palcomms.2017.78
Smith, K. E. & Stewart, E. (2017) ‘We need to talk about impact: why social policy academics need to engage with the UK’s research impact agenda’, Journal of Social Policy, vol. 46, no. 1, pp. 109–127. doi: 10.1017/S0047279416000283
Syed, J. & Davies, J. (2016) ‘Diversity in the authorship of journal articles and REF impact case studies: how are UK business schools shaping up?’ BAM2016 Conference Proceedings, [online], Available at: http://eprints.hud.ac.uk/id/eprint/29379/
Tusting, K. (2018) ‘The genre regime of research evaluation: contradictory systems of value around academics' writing’, Language and Education, vol. 32, no. 6, pp. 477–493. doi: 10.1080/09500782.2018.1505905
Tusting, K., et al., (2019) Academics Writing: The Dynamics of Knowledge Creation, Routledge, London.
UKRI. (2019) ‘How we fund research’, Research England website, [online] Available at: http://re.ukri.org/research/how-we-fund-research/
Vasant, S. (2014) ‘What is a learning technologist?’, Jobs.ac.uk website, [online] Available at: https://blog.jobs.ac.uk/education/teaching-learning/what-is-a-learning-technologist/
Watermeyer, R. (2012) ‘Issues in the articulation of ‘impact’: the responses of UK academics to ‘impact’ as a new measure of research assessment’, Studies in Higher Education, vol. 39, no. 2, pp. 359–377. doi: 10.1080/03075079.2012.709490
Watermeyer, R. & Lewis, J. (2017) ‘Institutionalizing public engagement through research in UK universities: perceptions, predictions and paradoxes concerning the state of the art’, Studies in Higher Education, vol. 43, no. 9, pp. 1612–1624. doi: 10.1080/03075079.2016.1272566
Wilkinson, C. (2019) ‘Evidencing impact: a case study of UK academic perspectives on evidencing research impact’,. Studies in Higher Education, vol. 44, no. 1, pp. 72–85. doi: 10.1080/03075079.2017.1339028
Woolridge, J. (2017) ‘Altmetrics linked to 3* and 4* impact scores in REF2014 impact case studies’, Altmetric blog, [online] Available at: https://www.altmetric.com/blog/altmetrics-ref2014-impact-case-studies/