The focus of this paper is the use of a field study to recover preliminary evidence, using the standardised measure of exam performance, that wiki-based exercises can enhance learning. Although strict causal inference is impossible with an observational study, we present an analysis which demonstrates that engagement with wiki exercises benefits both those who normally score highly and those who normally score less well on exams.

The structure for this paper is, first, an introduction to wikis and research on wikis in higher education. Secondly, we introduce the conditions of the field study, the methods of analysis and results. Finally, we discuss the interpretation of these results and offer speculation as to the cognitive mechanisms of action.

A wiki is a set of webpages which can be edited by readers. Wikis allow knowledge – and editing responsibility – to be shared by many people. The most famous example is, of course, Wikipedia (http://en.wikipedia.com), but many individuals and organisations run wikis large and small, public and private, for gathering and organising information.

Comprehension and retention of academic material is often assessed via students’ written work, whether under examination or coursework conditions. Although lectures and private study provide access to the information content covered by a course, as educators we are looking for more than simple absorption of this information among our students – we wish that they understand and are able to express that understanding lucidly in writing.

We reasoned that the use of a wiki to collect information on course topics, and as an arena for practicing the expression of those ideas, could augment students’ understanding. Wiki-based exercises would put the responsibility for collecting and structuring information about the topic on to the students. This active engagement with the course material should, we hypothesised, facilitate a change in students’ habits from mere memorisation of course material to deeper comprehension. Previous studies of concept learning have shown the beneficial effects of discussing material (Smith et al. 2009) and of generating explanations from expository text rather than merely reviewing it (Ainsworth and Th Loizou 2003; Chi et al. 1994; Coleman, Brown, and Rivkin 1997). Both discussion and reworking of material are encouraged by wikis, although – it should be noted – they consist of an extended, virtual, social environment, rather than a directly social one.

Such an approach is in line with a ‘constructivist’ approach to education (Bruner 1966), which emphasises the role of the learner in engaging with and reshaping to-be-learnt material during the learning process. More recently, constructivism has inspired the development of ‘Inquiry Based Learning’ (Edelson, Gordin, and Pea 1999; Stafford 2008), which puts student-led activities at the centre of the learning process. Wiki work has the advantage of being a shared activity which is closely aligned with what remains the primary outcome measure of higher education – an individual's ability to write on the course topic in answer to exam or coursework questions.

Wikis are part of a cluster of ‘Web 2.0’ technologies (O'Reilly 2005) which have arguably been responsible for a ‘participatory turn’ in higher education (Staley 2009, p. 36). These new technologies are not merely used to deliver content, but provide different ways for students and teachers to engage in the learning process. This review briefly discusses how wiki technology can restructure group and social relations in education, before turning to our primary interest of wikis as a structured environment for the development of written communication skills. In particular, we wish to highlight that previous research has not attempted to account for differing levels of engagement with wikis when investigating their putative benefits.

The flexible structure of wikis in particular offers a new format for collaboration (Duffy and Bruns 2006). A number of studies have investigated the impact of wiki use upon student collaboration during group work in higher education (Clougherty and Wells 2008; Duffy and Bruns 2006; Gokcearslan and Ozcan 2011; Morley 2012). More specifically, wiki use has been found to both improve the efficiency of group work (Clougherty and Wells 2008; Minocha and Thomas 2007) and the quality of interactions between students in a group (Kuteeva 2011). In addition, the giving and receiving of peer feedback via a wiki helped some students to begin a process of self-reflection and critical thinking that was apparently new to them: a student in one study explained, ‘Even though I understood exactly what I was trying to specify, it wasn't until I received feedback, and, indeed, gave feedback, that I realised that some of what I had written was open to misinterpretation’ (Minocha and Thomas 2007, p. 204). In this study, wikis therefore appeared to have the potential to help students recognise the value of collaboration. Other studies also concluded that wikis can improve the social cohesiveness of a student group (Duffy and Bruns 2006; Morley 2012). However, successful collaboration was not always achieved through wiki use. There is some evidence that difficulties arose when students presented different levels, or different kinds, of wiki participation. Wheeler, Yeomans, and Wheeler (2008) describe the phenomenon of social loafing, where some students hardly contribute at all. Other students were found only to write and edit their own pages, without attending to the contributions of others, or to be over-reliant on tutor assistance (May, Burgard, and Abbasi 2011; Naismith, Lee, and Pilkington 2011).

In addition to the largely positive picture the literature paints of wiki use for improving collaboration between students (and teachers), there is a good deal of attention given to the benefits of wikis for students’ writing. First, writing in a wiki sometimes appears to change the students’ understanding of themselves as authors. One paper describes this as ‘an interesting rethinking of the nature of authorship, and the sense that among some learners that the wiki space almost gives permission to “be” a writer in a new way’ (Hemmi, Bayne, and Land 2009, p. 28). Other papers have understood this change in perspective as part of the shift in the balance of power between students and tutors that are traditionally present (Lundin 2008; Minocha 2009); Ravid, Kalman, and Rafaeli (2008) consider wikis to be a tool for empowerment as students are actively participating and undertaking a ‘role traditionally assigned to a distant and disengaged expert’ (p. 1922).

The standard of students’ writing during wiki collaboration also appears to improve, namely in the areas of grammar, spelling, paragraph structure (Kuteeva 2011), clarity (Hemmi, Bayne, and Land 2009), accuracy and relevancy (Wheeler, Yeomans, and Wheeler 2008), and composition, structure and revision of writing (Pifarré and Fisher 2011). Some of these improvements are understood to be a result of students’ increased awareness of peer audience (Kuteeva 2011; Montero-Fleta and Pérez-Sabater 2011; Wheeler, Yeomans, and Wheeler 2008); as one student explained, writing ‘involves much more thought … as it is able to be read by anyone’ (Wheeler, Yeomans, and Wheeler, 2008, p. 993). A student from the same study said, ‘I am now developing a healthy critical and analytical writing style thanks to the wiki. Looking at other people's opinions and findings has helped me to question what's in front of me’ (Wheeler, Yeomans, and Wheeler 2008, p. 993).

Arguably, the role of wikis in encouraging critical thinking and self-reflection sits well with constructivist theories of learning (Minocha and Thomas 2007; Zorko 2009). Wikis highlight the importance of shared meanings: ‘the wiki becomes both a tool for choice and action and a flexible and extensible environment for collaborative negotiation of meaning’ and so follows the Vygotskian approach (Carr et al. 2007). However, the successful employment of wikis to catalyse the move from ‘knowledge representation’ to ‘knowledge construction’, argues Morley (2012), requires careful guidance or scaffolding by the tutor. Inadequate tutor involvement, support or explanation of how the wiki collaboration would work was commonly cited as a reason for difficulty (Cole 2009; Greenhow, Robelia, and Hughes 2009; May, Burgard, and Abbasi 2011; Morley 2012). Other problems noted with wiki use by students include being ‘locked out’ of editing when another student is using a page which can result in a loss of work (Duffy and Bruns 2006); frustration with the change in study pattern from last-minute essay writing, to sustained, lengthy participation (Minocha and Thomas 2007); technical problems (Morley 2012); ‘unproductive interactions’ and ‘threats to privacy’ (Greenhow, Robelia, and Hughes 2009, p. 251); and discomfort in the assessment process (Carr et al. 2007).

The current study sought to test the value of wiki use using an independent and highly regarded dependent measure – final exam performance. Whilst we might suppose, in line with the previous literature, that wiki use benefits collaboration, writing skills and comprehension of course material, this study allows us to assess the important ‘bottom line’: Are the benefits of wiki use substantial enough to have a measurable effect on student assessment? Although previous research has shown some evidence for a difference in final grades between those who do and do not use a course wiki (Laru, Näykki, and Järvelä 2012; Ravid, Kalman, and Rafaeli 2008) this research has not addressed the confound that more engaged and enthusiastic students are more likely to both use a course wiki more and to score well on the final exam. The present study is the first, to our knowledge, which does attempt to disentangle these factors.

Participants in the investigation were level 2 students (typically in their second year of a 3 year course studying for a BA or BSc in psychology) at the University of Sheffield, UK. The majority of undergraduate students in our department are female (83%) and UK nationals (90%). Ninety-three percent identify as White British. The investigation considered their use of a wiki and final exam grade as part of a cognitive psychology module, PSY241, which forms a core part of the undergraduate degree. As a core module the content is regulated by the British Psychological Society. Topics covered include memory, attention, reasoning, language and vision. Teaching focusses around an introductory cognitive psychology textbook (Andrade and May 2004), with 22 one-hour lectures and three practical classes taking place over 12 weeks. The final exam consisted of three essays from a choice of six, completed over 3 hours. It was graded on a 100-point scale, as are all other exams at the University. Over 2 years, we collected data from students who took the exam.

We used ‘media wiki’ (http://www.mediawiki.org) to create a course wiki. Only registered users could edit the wiki, a device which prevented vandalism and allowed us to track use of the wiki. The wiki was introduced in course lectures and initial exercises and optional on-going tasks were set to encourage engagement with the wiki. These exercises included writing content on the course topics, leaving questions for other students and proofreading. However, at no point was use of the wiki monitored during the course, nor was it implied to the students that use of the wiki was being monitored or was a part of the course assessment. Wiki use on the course was relatively unstructured and undirected by the course leader, with minimal formal integration of the wiki into the course, it was merely provided for student use. In other words, we left it open to individual students to decide on whether to use the wiki without hope of approval or fear of recrimination. This means that it is sadly not possible to give full details of how the wiki was used by each student. Although students were assigned the task of editing the wiki, and directed to focus on writing about the course material, they were not told what to write or how to organise the wiki beyond a skeleton structure based on the principle topics of the course. Students in the second year of the investigation were told that use of the wiki, according to the evidence available from the first year, was an effective study strategy.

The analysis was approved by the University of Sheffield, Department of Psychology Ethics Sub-Committee (DESC), and carried out in accordance with the University and British Psychological Society (BPS) ethics guidelines. The use and exam performance data was collected incidentally and so did not require any change in the teaching or behaviour of the students involved. For this reason, the institutional review board waived the need for written informed consent from the participants.

Data were collected from 216 students who took the examination over a 2-year period. The average grade for the students who took part in the study was 55.6 (standard deviation 10.5). For each student, it was also possible to collect their exam grades for the other modules they took in their second year of study (in the same semester and the semester after the cognitive psychology module). The average score on other level 2 psychology modules was 61.1 (standard deviation 7.23). This was roughly in line with previous years’ results.

We assessed wiki use by automatic logging of number of edits, the number of different articles edited and the number of different days on which the wiki was edited. A simple correlation between number of edits and final exam score was significant (Pearson's r=0.287, p<0.0001). The correlation between number of edits and average score on other second year modules was also significant (Pearson's r=0.589, p<0.0001). A regression, predicting PSY241 grade from number of wiki edits and average of other level 2 exam scores was significant (see Table 1) [R ²=0.367, F(2, 202)=58.633, p<0.0001].

In contrast, we also collected self-reported ratings for the first cohort of the study on seminar attendance and perceived usefulness of study seminars, along with reports of wiki use and perceived usefulness, and these had no significant predictive value for exam scores [R ²=0.071, F(4, 35)=0.592, p=0.671].

A moderation analysis was conducted to test for an interaction between wiki edits and average level 2 grade. The overall regression remained significant [R ²=0.368, F(3, 201)=38.975, p<0.0001], as did the contributions of the number of wiki edits and average second year grade, but the interaction between these two factors was not significant (weight = 0.009, b=0.034, t=0.0390, p=0.697). This suggests a consistent effect of wiki use on PSY241 grade across the range of average exam performance.

To illustrate these effects, we used a median split to divide the students according to their typical grade on other psychology courses, producing a ‘low’ and a ‘high’ group. We also categorised students according to their use of the wiki, non-user (0 edits, n=68), light users (1–3 edits, n=74) and engaged users (4 or more edits, n=74). The results can be seen in Figure 1.

Statistically, the beneficial effect of wiki use can be understood as the equivalent of engaged wiki use predicting a 4–5 percentage point gain on final exam score, which moves a student, for example, from a mid 2.1 grade to a first class grade in the standard British undergraduate degree classification system.

Analysis using the other indicators of wiki use – number of articles edited and number of days on which the wiki was edited – yielded similar results to the analysis of number of wiki edits. Additionally, a confirmatory factor analysis showed that a latent variable, composed of all the wiki use indicators, also remained an independent and additional contribution to PSY241 grade.

As expected, there is a correlation between students’ PSY241 exam grades and the average grade on other courses. Students who work harder and/or have greater academic skill will tend to get good grades on most courses. Those who work less hard generally and/or find academic work more difficult will tend to score worse on most courses. Although this study is a field study rather than an experimental design, the regression allows us to control for any general tendency to get good grades, and so to look specifically at the additional benefit of using the wiki. This is shown to be reliably and strongly associated with better exam performance. For both students who tended to get good grades, and those who tended to get less good grades, using the wiki provided a benefit in terms of final grade. Engaging with the wiki, defined as a mere four or more edits, was enough to gain students, on average, an extra 4–5 percentage points on the final exam. The results are all the more convincing because the self-paced, undirected act of writing for the wiki is very different from the time-pressured, question-focussed act of writing for the final exam. Our suspicion is that wiki use showed its benefit by acting on the most important hidden variable in pedagogy, comprehension.

Without doubt the quality of contributions to the wiki would also predict benefit, but we did not assess this, focussing on the raw count of number of edits made. As well as circumventing the tricky issue of how to assess quality of contribution, the focus on number of edits as an index of contribution keeps the emphasis of this investigation on wiki use as a beneficial process, rather focussing on the contents of the wiki edits. We choose to analyse wiki use from this ‘zoomed-out’ perspective, acknowledging that if wikis are found to benefit exam scores there is substantial work remaining to be done to characterise exactly how wikis are used and why they have any benefits.

That the current research is a field study, rather than an experimental design means we are able to take advantage of the high ecological validity of using real students engaged in a real learning task, and to use a real measure of learning (the final course exam). The disadvantages include that we have no experimental control and so cannot confidently ascribe causality to wiki use. We attempt to account statistically for hidden variables such as talent or enthusiasm which might create correlation between wiki use and exam performance, something which is novel in the existing literature on the benefits of wiki use in educational settings. Nonetheless, the ultimate test of the idea that wiki use benefits learning would be to run an experiment in which students were randomly assigned to either a wiki use condition or a control condition which was assigned a non-wiki task. Future investigations would also allow more detailed measuring of the nature of students’ engagement with the wiki so as to test exactly what aspects of wiki use lead to the exam performance benefits. As it is, the focus of our investigation means we can only develop informed speculation as to the likely cognitive mechanisms by which wiki use enhances learning. We now present a review of plausible mechanisms of action for wiki use on learning.

The potential for the use of wikis in education has not gone unnoticed (Hazari, North, and Moreland, 2009; Konieczny 2007) but this study provides direct evidence of the concrete benefits of wiki use to an acknowledged indicator of pedagogic outcome – namely final exam grades. We provide an example of how wiki use can be integrated into an existing course, and with minimal oversight and little emphasis on explicit collaboration can enhance both the student experience and learning outcomes. Additionally, the use of wiki technology provides students with a skill with wide application beyond that of the original course and programme of study. Future work should focus on both the pedagogic conditions which allow the benefits of wikis and isolate the cognitive mechanisms of action.