What constitutes good research in e-learning — are there lessons we can draw from the Research Assessment Exercise ?

The next Research Assessment Exercise (RAE) is nearly upon us and as such I thought it would be useful to reflect on what the RAE offers us in terms of positioning our own research discipline and how it might help us to answer the question ‘what constitutes good e-learning research?’ DOI: 10.1080/09687760701673576

The next Research Assessment Exercise (RAE) is nearly upon us and as such I thought it would be useful to reflect on what the RAE offers us in terms of positioning our own research discipline and how it might help us to answer the question 'what constitutes good e-learning research?'

Overview of the RAE
The RAE is intended to provide a periodic review of the quality of research in higher education institutions, and is important not only because it essentially results in a 'research league table' but also because significant funding is attached to the process for those areas deemed to be research excellent.A newspaper article last year (Guardian Unlimited, 2006) highlighted the central role that the RAE has played in the direction and focus of UK research activities and associated culture: Over two decades the RAE has become an obsession for British academics and the ratings-from one to five-star-have made or broken the reputations of university departments.On the basis of the RAE judgments made at roughly five-year intervals depend billions of pounds of funding from the Higher Education Funding Council for England and its equivalents in Scotland, Wales and Northern Ireland.
The exercise is designed to be peer reviewed by a panel of experts drawn from the relevant research community.There are 67 units of assessment.Each academic submitted is judged on a number of criteria.The most important of these is the four publications (journal articles, books, etc.) they choose to submit as evidence of their academic standing published during the review period.Gradings in the new system range from zero to four (Table 1).In addition, each person submits an 'indicator of esteem' that includes information on their success in research funding, involvement as referees or journal editors, invited talks and keynotes, or other recognised evidence of academic worth and 'international standing'.

Criticisms of the exercise
Despite the fact that the exercise is peer reviewed there have been significant concerns voiced over its validity.Firstly, many argue that the quality across the panels varies significantly, with wide disputes and differences in what is deemed 'high-quality research'.Others feel aggrieved that money-rich disciplines in their opinion fair much better; the new post-1992 universities make similar arguments, and it is true that they have faired less well overall.And it is widely known that one of the purposes of the introduction of the exercise was to make research-funded institutions more selective, with money biased towards the university elite.Many also argue that the overconcentration on the RAE and its importance has further deepened the divide between research and teaching, with teaching now more than ever suffering as the poor sister.For individuals in research-intensive institutions, deciding where to concentrate their efforts is simple; research wins every time … if you want to get promoted that is.

Lessons from the RAE
This is the third RAE exercise with which I will have been involved.The first was in my original academic area, chemistry.In a sense submission then was fairly straightforward for me-I published in the standard recognised journals for my area, and I was lucky enough to be collaborating with a good range of internationally recognised researchers.When I moved into the area of e-learning, things became less clear; as a relatively young research area should I be publishing in new e-learning-focused journals or more mainstream, well-established education journals?What should the balance be in terms of standard empirically based submissions and more risky, but perhaps more innovative, approaches?
I have been involved in helping with the RAE mock exercises at my current institution (the Open University) and my previous institution, Southampton University, both in terms of developing research group narratives and peer-reviewing exercises.It has been interesting and educational but reiterates for me the extremely subjective (i) There is evidence of novel, new thinking, new approaches.(j) There is evidence that the findings are having impact beyond one institution.(k) There is a clear articulation of the methodology and a critique of the approach adopted.(l) The submission demonstrates evidence of linking to higher agendas of the daypolicy directives or evidence of ideas being embedded in or aligned with funding council programmes.(m)It is a retrospective piece showing how the work builds on or provides a foundation for other work that followed.(n) The paper provides clarity and insight into a well-recognised problem.
On reading the papers, a raft of unresolved issues came to mind: (a) In such a fast-moving area as e-learning, is there an issue with papers becoming dated; what papers 'age' well and what are their characteristics?(b) Academics submit four artefacts; these are supposed to be reviewed separately, but how much importance will the panel actual put in the coherence of the set in terms of the breadth of expertise demonstrated by researcher across four papers chosen?Will overlapping submissions be viewed unfavourably?(c) What issues are there around individual contributions and multiple authorships?
Is collaboration good or bad?Do international co-authors add benefit?(d) How will the panels really view other outputs of research-for example, a web site of resources or a technological artefact?(e) How is the issue of comparability going to be addressed?One submission may represent five years of work, another a small-scale study.(f) One could argue that, if you take a very strict reading of the criteria in Table 1, very rarely is work 'seminal' (i.e.four star)-but what does this mean in terms of the message it sends on the perceived worth of UK research?(g) How a paper is viewed inevitably depends on who is reading the paper-if it is an expert in the field they either may be more critical (because they are very aware of the subtleties of the work and/or may have a different take on the area) or interested (because it aligns with their own work), compared with a researcher from another area who might rate a review paper highly because it provides an overview/summary of the area or might be disinclined to a paper because it adopts a radically different epistemological position than their own.
The RAE is a mammoth exercise, which has absorbed enormous amounts of time and resources, for questionable benefit.RAE 2008 will be the last exercise carried out in this painstaking and complex process.The Government has announced that the exercise will be replaced by a much lighter weight, metrics-based system.However it is not clear what this will involve, and no doubt a barrage of complaints will surround its introduction as well.It seems it is impossible to get the balance right.While I do not think many in academia would argue against some form of peer-assessment and recognition of worth (after all surely this is the core of how we work anyway through the peer-review funding and publication mechanisms), somehow how it has been done to date does not seem to have been thought through enough.I would advocate for a more interactive, formative and constructive approach with feedback on a timely basis to academics in terms of helping them maximise their research potential whilst balancing this in terms of ensuring high-quality teaching, with strong synergies between the two.

The RAE and e-learning
As a recent book on contemporary perspectives in e-learning research (Conole & Oliver, 2007) notes, e-learning is a complex field of inquiry: This is potentially both a strength and a weakness of our area.There is no unit of assessment for e-learning; instead, researchers will be returned via the nearest cognate area such as education or computer science, but how will the assessors on these panels view the e-learning submissions?
How do the papers in this issue of ALT-J, for example, fair in relation to the criteria discussed above?It certainly struck me reading the papers that they represent very diverse research, using different methodologies and approaches.For example, the paper by Barnes and Tynan is not a traditional empirically based paper.Instead, they describe a brave new world of the near-future, outlining how students might appropriate technologies to support their learning.I believe the paper is important because it raises a raft of questions and issues about the impact of the vision of the (near) future for both us as teachers in how we design learning activities and provide support for this new generation of students and how institutions provide an appropriate and enabling infrastructure.The next two papers provide more traditional accounts.Downing et al. report on the findings from a case study looking at online interaction.

Editorial 187
The paper builds nicely on the broader body of research in this area over the past decade, looking at trying to understand how students learn and interact online.Villar and Alegre also report on findings from an empirical study, but their focus is on the evaluation of an online staff development system designed to improve teachers' pedagogical knowledge and teacher practice.Again, this work aligns with a wide body of research that has been exploring the changing role of teachers and mechanisms for providing them with the skills necessary to use e-learning.Accessibility is an important and active area of e-learning research, and Cooper et al. argue that good practice drawn from accessibility research should be integral to any e-learning research and development project.They draw on their experiences from across a range of case studies.Finally, Jones and Isrroff provide a theoretical paper, which considers two different theoretical approaches-technology appropriation and coping strategies-as a means of understanding motivational factors in relation to the use of mobiles for learning.Importantly they argue that research in this field is under-theorised, and hence the paper provides a valuable attempt to address this.The five papers in the present issue are very, very different in focus, methodological approaches and 'data'/findings.It might be an interesting exercise to try and apply the criteria I outlined above to these papers and see what they tell us in terms of the papers' research quality.I suspect each of us would have different opinions in terms of the merits of each paper and its 'research rating'-which begs the question, is there any point in such an assessment exercise?I am stepping down as editor; this will be my last issue.It has been an interesting experience-certainly I remember what a huge learning curve it was taking over the first issue with which I was involved, as deputy editor for ALT-J 6(3) back in 1998, and production of each issue always seems to be an agonising and painstaking experience that does not get any easier with time!However, I have learnt a huge amount in the process and being editor has enabled me to gain a perspective on the way elearning as a research area has developed.I was lucky to have the privilege to work alongside Professor David Squires, until his untimely death in August 2001-issue 10(3) is dedicated to him and contains research articles from colleagues from around the world who worked with him.In the past nine years the nature of the content in ALT-J has changed, reflecting the general changes in e-learning more broadly.Martin Oliver, Jane Seale and myself as the editorial team have worked hard to improve the quality of the articles published in the journal and have instigated a more rigorous refereeing process and better feedback to authors on how their papers might be improved.We have attempted to introduce a more discursive note to the journal through the inclusion of discussion pieces.We have worked hard also at raising the profile of the journal internationally; our move to Taylor & Francis, the membership of our editorial board and editorial presentations at conferences have all helped in this respect, and it is encouraging to see papers from Spain and Australia in this current issue.I look forward to seeing the journal continue to improve and to being a relaxed, reader, rather than a hassled editor!Professor Gráinne Conole Editor

Table 1 .
Definitions of quality levelsFour starQuality that is world-leading in terms of originality, significance and rigour Three star Quality that is internationally excellent in terms of originality, significance and rigour but that nonetheless falls short of the highest standards of excellence Two star Quality that is recognised internationally in terms of originality, significance and rigour One star Quality that is recognised nationally in terms of originality, significance and rigour.Unclassified Quality that falls below the standard of nationally recognized work.Or work that does not meet the published definition of research for the purposes of this assessment nature of the process.Here are a list of some of my thoughts when reading and rereading articles.I know from talking to other colleagues involved in the process that many others echo similar concerns.The following is a list of some of potential 'indicators of worth' for a paper: It has a good grounding in the literature and evidence of knowledge of key issues.