Examining educational technology and research impact: the two roles of e-learning and related terms in the 2014 REF impact case studies

Research in Learning Technology 2020. © 2020 K. Jordan. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.


Introduction
The issue of 'research impact' is of critical importance to the Higher Education sector in the UK and is dealt with by using the research excellence framework (REF). The REF is a national audit of research outputs, and its result has implications for the levels of funding that institutions will receive. The REF was launched in 2014, replacing a previous model known as the research assessment exercise (RAE). One of the key distinctions between the REF and the RAE was the foregrounding of the perceived 'real world' impact of research, framed as a reflection of better 'value' for research funding (Jump 2013). As a result, the 2014 REF was the first assessment exercise to include submissions of 'impact case studies' as a mechanism for representing and assessing evidence of impact. Whilst such impacts may be challenging to define and it is debatable whether the case study format accurately captures impact (Khazragui and Hudson 2015), academics do perceive its inclusion to be valuable in presenting a richer account of scholarly activity (Watermeyer 2012 In the 2014 REF, 6975 case studies were submitted, 6679 of which were subsequently published online in a database (REF 2014). The database represents a valuable resource for further research and analysis in relation to research impact. As 2014 was the year of the introduction of REF, the sector is actively learning and adapting to this new form of output (Tusting et al. 2019), and the database has been a source of information for several cross-cutting analyses to highlight the role of impact case studies in practice.
The importance of impact case studies has been underlined by quantifying the link to funding received by institutions. Although the database does not include the results of the assessment for each case study, information may be inferred from the average scores of units of assessment. Reed and Kerridge (2017) examine the link between units of assessment which obtained a score of 3* or 4* (the highest grades conferred by the REF) for their impact case studies in the 2014 REF and their subsequent QR funding allocation in 2016/2017, concluding that on average a 4* case study was worth approximately £35000 more than a 3* one in funding terms.
Trends in the database have also been examined in order to characterise what constitutes an impact case study. King's College London and Digital Science (2015) used text-mining to consider over-arching trends in the database, coupled with addressing specific questions identified as being of key importance through workshops. The report notes that impact case studies are often multi-disciplinary in nature, and that different institutions appear to specialise in different types of impact to a certain extent (ibid). Further characterisation of trends in the database overall has interrogated the nature of impact within the case study format by examining the relationships between the cases, public engagement and traditional scholarly metrics. Public engagement has been shown to be frequently linked to impact case studies, but there is variation in the prevalence and form of public engagement across different disciplines (Duncan and Manners 2017;Watermeyer and Lewis 2017). Contrasting findings have been reported about the relationship between impact case studies and scholarly bibliometrics. With the addition of data about the scoring received by case studies in the REF, Woolridge (2017) demonstrates a link between highly rated case studies and altmetrics scores, while Ravenscroft et al. argue that the link between impact case studies and traditional citation-based metrics is weak (Ravenscroft et al. 2017). Social media also plays a role, which has been mentioned in approximately one quarter of the case studies (Jordan and Carrigan 2018).
While case studies are often multi-disciplinary (Digital Science 2016), the nature of impact varies according to discipline and subject area. In examining the types of evidence submitted in support of impact case studies, Loach, Adams and Szomszor (2016) report testimonials, reports and articles as being the most frequently used, and some disciplinary preferences emerged. For example, testimonials were the most frequently used evidence type, and used to a significantly greater extent in arts and humanities, while under-used overall in biological sciences and medicine (ibid). The converse is true of reports and articles, which are the second and third most frequently used evidence sources (ibid). Subjects also vary in the extent to which the papers submitted as supporting evidence in case studies are also submitted to the main body of the REF, with more applied 'units of assessment' demonstrating greater overlap (Digital Science 2016). Several studies have examined REF impact case studies in specific subjects, for example, from the perspectives of anthropology (Simpson 2015), business (Syed and Davies 2016), educational research (Cain and Allan 2017;Cotton, Kneale, and Miller 2018), engineering (Biri, Oliver, and Cooper 2014;Robbins, Wield,

(page number not for citation purpose)
and Wilson 2016), health (Greenhalgh and Fahy 2015;Hinrichs and Grant 2015;Kamenetzky et al. 2016;Kelly et al. 2016), leadership, governance and management (Ross and Morrow 2016), library and information science (Marcella, Lockerbie, and Bloice 2016) and social work (Smith and Stewart 2017). As the 2014 REF was the first to make use of the impact case studies format, different subject areas have sought to learn lessons from the database about the character of impact case studies in their subjects to reflect on the nature of impact and prepare for the next REF, when the weighting of case studies will be increased from 20% to 25% (Else 2017).
There is a gap in the current literature in relation to educational technology and research impact. Examining the role that educational technology and related terms played in the 2014 REF, impact case studies can provide useful insights and practical implications for the field in two main ways. Firstly, they help in understanding how research into educational technology and related topics were located within the 2014 REF. This is a particularly important question as technology-enhanced learning is an interdisciplinary field (Conole et al. 2010). Secondly, they provide insights into the ways in which educational technologists can support academics and practitioners from across the disciplines in relation to developing their REF submissions. The following questions guided the analysis presented here: • How prevalent were educational technology-related terms in the 2014 REF impact case studies? • Which types of case studies are educational technology-related terms associated with, in terms of subject areas and types of impact? • What role do educational technology-related terms play in relation to impact case studies?

Methods
Data were sourced from the 2014 REF impact case studies database, which is a freely available online database (REF 2014). The database comprises 6679 non-redacted case studies, which represents 96% of the total 6975 case studies submitted to the 2014 REF (ibid). Database searches were conducted using the following search terms (Table 1). While there are many more search terms related to specific educational technologies or pedagogic innovations that could have been included, the terms were selected to ensure that the main synonyms for educational technology were covered. This strategy was adopted in order to maintain a focus on the role of educational technology in general.: Information about the case studies returned by each search was exported as a series of spreadsheets. The spreadsheets were combined, duplicates removed and each case tagged with the search terms associated with it. Two cases were later removed due to false positive searches. Both cases were associated with the 'learning technology' search term, and both had been found due to the term 'machine learning technology' being present in the case study. The resulting sample comprised 125 case studies. Since the REF 2014 impact case studies database is a publicly available online resource, no ethical issues were associated with the work. However, although the cases do include names and other personally identifiable data including institutional affiliation, it would not add anything to the research questions at hand to include these data types here. Once the sample had been constructed and prepared, each case was examined in turn by the researcher to identify where the corresponding search term was found, and to categorise the ways in which educational technology and the related terms were used. As the study was exploratory in nature, an open coding approach to categorisation was used (in the manner of grounded theory, but short of a full grounded theory-based design; Charmaz 2014; Glaser and Strauss 1967), rather than imposing a pre-existing coding scheme. The possibilities of basing a coding scheme on an existing typology of educational technologies or types of pedagogical context or approach were considered; however, neither were used, for two reasons. First, there was considerable variation in the level of detail available in the case studies, which would have posed a challenge for applying pre-existing typologies. Second, neither potential typology would have fully encompassed the focus of the research question here (the coding being used particularly to address research question 3), which is concerned with the role that the terms are playing in this context, for which there is no existing coding scheme. On this basis, open coding was used, to support the emergence of categories from the sample.
In coding the cases, two major distinct groups within the dataset quickly became apparent, and a sense of theoretical saturation emerged after approximately 25 cases (Morse 2007). Ultimately, analysis of the coded dataset was descriptive in nature, to examine trends in relation to subject area, types of impact and how educational technology-related terms are being used in this context. Links between categories within subject areas, impact types and the use of terms are presented in the analysis in a form derived from Sankey diagrams. While Sankey diagrams are a tool intended to represent the flow of resources within complex systems, typically manufacturing, the format has been co-opted by the field of data visualisation and can be applied to sets of categorical data more broadly (Bendix, Kosara, and Hauser 2006;Lupton and Allwood 2017).

Research question 1
The first research question considered 'How prevalent were educational technologyrelated terms in the 2014 REF impact case studies?'. The sample included 125 unique case studies, which represents 1.9% of the total 6679 cases in the database. While this may suggest that educational technology and related terms are relatively niche within this context, 125 is not an insubstantial number. The number of case studies returned per search term are shown in Table 1. The distribution shows that the sample is dominated by use of the term 'e-learning', which alone accounts for approximately half of the case studies. It should be noted that the figures reflect the numbers per query, prior to de-duplication. A total of 34 of the cases (27.2%) contained two or more of the search terms; 11 (8.8%) contained between three and five. Visualising the information as a network graph is a useful way to explore patterns within those containing multiple search terms, as it allows multiple relationships between terms to be mapped. The co-occurrence of search terms within the de-duplicated sample is shown as a weighted network (Ackland 2013) rendered using Gephi ( Bastian, Heymann, and Jacomy 2009) in Figure 1. In a weighted network, the links ('edges') between terms are assigned values based on the strength Figure 1. A network graph illustrating the co-occurrence of educational technology-related terms in the sample. Edges are weighted to reflect the number of cases which contain a pair of terms; node sizes reflect the number of cases which include a particular term. Edge weights are shown in Table 2 of the connection (Ackland 2013). The numbers of cases that included the pairs of search terms -and thus reflect the thickness of the edges in Figure 1 -are shown in Table 2. E-learning emerges as the most prominent term, most frequently associated with 'elearning', 'educational technology' and 'technology-enhanced learning'. These characteristics suggest that there may be two distinct sub-groups within the sample; a majority which simply refer to 'e-learning', and a minority which use multiple ways of discussing educational technology.

Research question 2
The second question was concerned with examining the types of case studies in which educational technology-related terms are found. The database readily provides information about the type of impact, and subject area ('unit of assessment'). While information about the submitting institution is also available, it was not included in the analysis because the number of case studies submitted varies by institution size, so trends may be misleading.
The number of cases included in the sample according to type of impact are shown in Table 3. As the total number of cases submitted illustrating different types of impact varies substantially (from 212 demonstrating 'legal' impact to 1723 demonstrating 'societal' impact), the figures are also given as a percentage of the total number in the database per impact type. Note that the typology of impact types is defined by the REF impact case studies database. Each case study within the database states the associated impact type in its introductory text (REF 2014). The cases containing educational technology-related terms are mainly associated with two types of impact: 'societal' and 'health'. Table 4 shows the representation of different subject areas (or 'units of assessment', to use the categorisation scheme adopted by the REF) within the dataset. For the REF, units of assessment are arranged into four broader disciplinary groups, or 'panels'. The sample includes cases drawn from all four panels, to different extents. Panel B, which focuses upon the natural and physical Sciences, is least well-represented. Within the other panels, there are substantial differences according to unit of assessment. Three units stand out as having greater than 5% of their cases mentioning educational technology-related terms. These are: allied health professions, dentistry, nursing and pharmacy; computer science and informatics and education.

Research question 3
In order to address the third question, 'what role do educational technology-related terms play in relation to impact case studies?', the text of each case study was examined to identify and categorise how the search term was being used. Two distinct, over-arching categories emerged from the data: those in which educational technology was an explicit part of the focus of the underpinning research in the case study (57), and those in which educational technology had been used to facilitate dissemination of other research findings (65). Additionally, three case studies included the term 'e-learning' as part of a testimonial contributors' job description. The two categories are clearly associated with a contrasting range of impact types (Figure 2). Furthermore, the categories are also associated primarily with different panels, with those where educational technology is used only for dissemination being mainly associated with panel A (Figure 2).
The group of cases where educational technology was simply used as a route to achieving impact (referred to as 'outputs' for brevity in Figure 2) was quite homogenous, typically including a statement that an e-learning module or materials had been made available to disseminate the findings of the research. The type of impacts documented by the cases in this group included cultural (7), economic (2), environmental (2), health (24), legal (1), political (5), societal (19) and technological (5). The profile of the cases in this group according to unit of assessment is shown in Table 5. All but five of the 65 cases in this group were included on the basis of the keywords 'e-learning' or 'elearning'. As such, this group of cases were not subject to further analysis.
For the 57 cases where educational technology was the focus of the research underpinning the case study itself (referred to as 'research context' in Figures 2 and 3), the type of impact being reported in this group was predominantly societal (46), and also cultural (5), health (1) and technological (5). The distribution of the 57 cases according to unit of assessment is shown in Table 6. Note that in comparison to Table 5, there is less variation in subject areas here, as the cases within this group are more frequently found within either the computer science or education panels.
A second round of finer-grained classification was undertaken on the 57 cases in which educational technology was part of the focus of the research itself. As the scope   Figure 2. Relationships between the two main categories of impact case studies identified within the sample (shown in the middle), the type of impact presented by the cases (left) and the panels the cases fall within (right). Note that the three cases where 'e-learning' was only included as a testimonial affiliation are not shown.
of research being reported in case studies can vary from an individuals' career, to a particular project, group or programme of research, this subsequent categorisation proved more challenging. The cases required two rounds of categorisation but nonetheless could be broadly divided into three categories based mainly on the positioning of educational technology in relation to the research context at hand. The following categories were identified, with the number of case studies contributing to each shown in brackets: • Development and evaluation of subject-specific educational technology initiatives (21; 36.8%): This was the largest group within the sub-sample, and represents a more applied group of research settings. In this group, the role of educational technology was more elaborate than simply dissemination (as in the other sub-sample) but focused on deploying educational technologies in a range of different subject settings. • Technological and app development (14; 24.6%): Cases where the focus is principally upon the development of technologies or apps that are related to educational technology, such as developing collaborative video technologies, technical specifications for e-learning or apps for citizen science.
As noted, the range of research covered by an impact case study can be broad. Cases were assigned to the category that best described their positioning in relation to educational technology and related terms, but a case study may have had links to several categories. The four sub-categories of case studies are also associated with different profiles in relation to the impact types reported, and submission panel (Figure 3). The number of case studies per sub-category and panel are shown in further detail according to unit of assessment in Table 7. Subject-specific educational technology initiatives span a wider range of subject areas and impact types. The categories of educational technology-focused educational research and technological and app development are both broadly split between panels C (education unit of assessment) and B (computer science unit of assessment). However, a key distinction is that while educational technology-focused educational research is associated exclusively with societal impact, technological and app development also includes technological and cultural impacts.

Discussion and conclusions
Across the levels of analysis presented here, educational technology and related terms played two distinct roles in the 2014 REF impact case studies. The sample of case studies from the REF impact case studies was approximately equally split between the two roles. On the one hand, educational technology-related research is being showcased through submission of REF impact case studies. These cases are associated with a wider range of educational technology-related terms, where the impact type is mainly 'societal' and principally found in cases assigned to the education unit of assessment (and to a lesser extent, computer science and informatics). On the other hand, educational technology plays a role in cases as a route to achieving impact. This is often framed in terms of 'e-learning' provision and associated with cases across a range of units of assessment, but notably to a greater extent in health professions and medicine. Both characterisations have practical implications for learning technologists and their work within higher education institutions. Definitions of the term 'learning technologist' encompass a wide range of activities mediating between technologies, staff and students (ALT 2018;Vasant 2014). The influence of the REF has led to Table 6. The number of cases categorised as featuring educational technology-related terms in relation to 'research context' (total n = 57), according to the unit of assessment they were submitted to. impact case studies becoming distinct emergent types of academic literacies (Tusting 2018;Tusting et al. 2019;Wilkinson 2019). If, as the analysis suggests, educational technology is being incorporated into this type of discourse, this is an area where learning technologists may require further training. For example, impact in the context of the REF is assessed in terms of both 'reach' and 'significance' of changes to practice (REF 2018). Simply making an e-learning module available through an online network with a known number of users may be an effective way of achieving reach, but this is only half of the picture. However, simple mechanisms for capturing significance (such as user surveys) could easily be incorporated at the design stage.
In terms of the potential for educational technology to enhance impact of research in general, the results suggest that educational technology may be under-used in relation to the REF and research impact at present, and that learning technologists could play an important role in further supporting this within institutions. Educational technology as a route to achieving impact was predominantly used in health and medicine-related subjects, as this is an area for which e-learning is already embedded for ongoing training and continuous professional development. Although the infrastructure and practices around e-learning are already established in this field, developing e-learning modules could be deployed as an impact strategy in any field, particularly

(page number not for citation purpose)
The findings also shed light upon the role of educational technology as a research field in itself. As the REF is a high-profile institutional exercise but has only been carried out once to date, the field is still in a nascent stage and adapting to the new format of impact case studies, and it is potentially useful to explore and reflect upon where educational technology-related research sits within this. Educational technology in relation to research impact in this context spans a range of roles between being the  Cotton et al. 2018). As cases that report educational technology-related educational research are divided between the Education and Computer Science units of assessment, further analysis of the computer science cases would be useful in order to contrast with the existing work in education to see if there are implications associated with which unit cases are submitting to. REF impact case studies and their associated literature are an area in which the role of professional networks of learning technologists, such as those facilitated by the Association of Learning Technology, could be particularly valuable as a way of supporting practitioners and promoting knowledge exchange between institutions. Given the competitive nature of the REF and its direct link to funding (UKRI 2019), institutions may be reluctant to share best practices, but would likely appreciate the opportunities for staff development, so this is an area where professional networks could provide support. Additionally, the cyclical nature of REF assessment periods mean that preparations intensify in the run-up to submissions. Personnel and expertise from one round may be lost in the years immediately following one of the exercises.

Limitations
This study is limited to an extent in that it is exclusively focused upon the UK. While REF impact case studies are a high-profile issue within the Higher Education sector in the UK, the model has not yet seen uptake in other countries (Sivertson 2017). One exception can be found in the Australian Higher Education sector, which has recently seen the introduction of the engagement and impact assessment, in addition to its main RAE (Havergal 2019). However, increasing the impact of research beyond academia is a desirable outcome whether it is formally audited or not, and a key issue for the learning technology community to engage with.

Conclusions
This exploratory analysis has shown that educational technology and related terms are feeding into REF impact case studies. While the number of cases is small when seen as a percentage of the whole database (1.9%), a substantial number (125) do contain educational technology-related terms. The cases are divided between instances where e-learning has been deployed as a route to achieving impact, and those where educational technology is the focus of the research. The use of educational technology as a tool to promote impact (by developing e-learning modules to communicate the findings of a research project, for example) is principally associated with subjects within the biological and medical sciences. The use of educational technology-based tools as a means of dissemination and enhancing impact may be useful for other disciplines and subject areas to adopt. However, simply developing tools is not sufficient to demonstrate impact in terms of the REF. Development for learning technologists may be required, to raise awareness of how e-learning tools could be designed in order to capture significance and reach. Educational technology-focused case studies are most frequently associated with education and computer science units of assessment and classed as demonstrating 'societal' impact. Further detailed analysis of this group of cases would be valuable in order to gain an in-depth understanding of how impact is achieved and demonstrated in this context.