Students’ engagement with a collaborative wiki tool predicts enhanced written exam performance

We introduced voluntary wiki-based exercises to a long-running cognitive psychology course, part of the core curriculum for an undergraduate degree in psychology. Over 2 yearly cohorts, students who used the wiki more also scored higher on the final written exam. Using regression analysis, it is possible to account for students’ tendency to score well on other psychology exams, thus statistically removing some obvious candidate third factors, such as general talent or enthusiasm for psychology, which might drive this correlation. Such an analysis shows that both highand low-grading students who used the wiki got higher scores on the final exam, with engaged wiki users scoring an average of an extra 5 percentage points. We offer an interpretation of the mechanisms of action in terms of the psychological literature on learning and memory.


Introduction
The focus of this paper is the use of a field study to recover preliminary evidence, using the standardised measure of exam performance, that wiki-based exercises can enhance learning. Although strict causal inference is impossible with an observational study, we present an analysis which demonstrates that engagement with wiki exercises benefits both those who normally score highly and those who normally score less well on exams.
The structure for this paper is, first, an introduction to wikis and research on wikis in higher education. Secondly, we introduce the conditions of the field study, the methods of analysis and results. Finally, we discuss the interpretation of these results and offer speculation as to the cognitive mechanisms of action.

Wikis
Awiki is a set of webpages which can be edited by readers. Wikis allow knowledge Á and editing responsibility Á to be shared by many people. The most famous example is, of course, Wikipedia (http://en.wikipedia.com), but many individuals and organisations run wikis large and small, public and private, for gathering and organising information.
Comprehension and retention of academic material is often assessed via students' written work, whether under examination or coursework conditions. Although lectures and private study provide access to the information content covered by a course, as educators we are looking for more than simple absorption of this information among our students Á we wish that they understand and are able to express that understanding lucidly in writing.
We reasoned that the use of a wiki to collect information on course topics, and as an arena for practicing the expression of those ideas, could augment students' understanding. Wiki-based exercises would put the responsibility for collecting and structuring information about the topic on to the students. This active engagement with the course material should, we hypothesised, facilitate a change in students' habits from mere memorisation of course material to deeper comprehension. Previous studies of concept learning have shown the beneficial effects of discussing material (Smith et al. 2009) and of generating explanations from expository text rather than merely reviewing it (Ainsworth and Th Loizou 2003;Chi et al. 1994;Coleman, Brown, and Rivkin 1997). Both discussion and reworking of material are encouraged by wikis, although Á it should be noted Á they consist of an extended, virtual, social environment, rather than a directly social one.
Such an approach is in line with a 'constructivist' approach to education (Bruner 1966), which emphasises the role of the learner in engaging with and reshaping to-belearnt material during the learning process. More recently, constructivism has inspired the development of 'Inquiry Based Learning' (Edelson, Gordin, and Pea 1999;Stafford 2008), which puts student-led activities at the centre of the learning process. Wiki work has the advantage of being a shared activity which is closely aligned with what remains the primary outcome measure of higher education Á an individual's ability to write on the course topic in answer to exam or coursework questions.
Previous research on the use of wikis in further and higher education Wikis are part of a cluster of 'Web 2.0' technologies (O'Reilly 2005) which have arguably been responsible for a 'participatory turn' in higher education (Staley 2009, p. 36). These new technologies are not merely used to deliver content, but provide different ways for students and teachers to engage in the learning process. This review briefly discusses how wiki technology can restructure group and social relations in education, before turning to our primary interest of wikis as a structured environment for the development of written communication skills. In particular, we wish to highlight that previous research has not attempted to account for differing levels of engagement with wikis when investigating their putative benefits.
The flexible structure of wikis in particular offers a new format for collaboration (Duffy and Bruns 2006). A number of studies have investigated the impact of wiki use upon student collaboration during group work in higher education (Clougherty and Wells 2008;Duffy and Bruns 2006;Gokcearslan and Ozcan 2011;Morley 2012). More specifically, wiki use has been found to both improve the efficiency of group work (Clougherty and Wells 2008;Minocha and Thomas 2007) and the quality of interactions between students in a group (Kuteeva 2011). In addition, the giving and receiving of peer feedback via a wiki helped some students to begin a process of self-reflection and critical thinking that was apparently new to them: a student in one study explained, 'Even though I understood exactly what I was trying to specify, it wasn't until I received feedback, and, indeed, gave feedback, that I realised that some T. Stafford et al. of what I had written was open to misinterpretation' (Minocha and Thomas 2007, p. 204). In this study, wikis therefore appeared to have the potential to help students recognise the value of collaboration. Other studies also concluded that wikis can improve the social cohesiveness of a student group (Duffy and Bruns 2006;Morley 2012). However, successful collaboration was not always achieved through wiki use. There is some evidence that difficulties arose when students presented different levels, or different kinds, of wiki participation. Wheeler, Yeomans, and Wheeler (2008) describe the phenomenon of social loafing, where some students hardly contribute at all. Other students were found only to write and edit their own pages, without attending to the contributions of others, or to be over-reliant on tutor assistance (May, Burgard, and Abbasi 2011;Naismith, Lee, and Pilkington 2011).
In addition to the largely positive picture the literature paints of wiki use for improving collaboration between students (and teachers), there is a good deal of attention given to the benefits of wikis for students' writing. First, writing in a wiki sometimes appears to change the students' understanding of themselves as authors. One paper describes this as 'an interesting rethinking of the nature of authorship, and the sense that among some learners that the wiki space almost gives permission to ''be'' a writer in a new way' (Hemmi, Bayne, and Land 2009, p. 28). Other papers have understood this change in perspective as part of the shift in the balance of power between students and tutors that are traditionally present (Lundin 2008;Minocha 2009); Ravid, Kalman, and Rafaeli (2008) consider wikis to be a tool for empowerment as students are actively participating and undertaking a 'role traditionally assigned to a distant and disengaged expert ' (p. 1922).
The standard of students' writing during wiki collaboration also appears to improve, namely in the areas of grammar, spelling, paragraph structure (Kuteeva 2011), clarity (Hemmi, Bayne, and Land 2009), accuracy and relevancy (Wheeler, Yeomans, and Wheeler 2008), and composition, structure and revision of writing (Pifarré and Fisher 2011). Some of these improvements are understood to be a result of students' increased awareness of peer audience (Kuteeva 2011;Montero-Fleta and Pérez-Sabater 2011;Wheeler, Yeomans, and Wheeler 2008); as one student explained, writing 'involves much more thought . . . as it is able to be read by anyone' (Wheeler, Yeomans, and Wheeler, 2008, p. 993). A student from the same study said, 'I am now developing a healthy critical and analytical writing style thanks to the wiki. Looking at other people's opinions and findings has helped me to question what's in front of me' (Wheeler, Yeomans, and Wheeler 2008, p. 993).
Arguably, the role of wikis in encouraging critical thinking and self-reflection sits well with constructivist theories of learning (Minocha and Thomas 2007;Zorko 2009). Wikis highlight the importance of shared meanings: 'the wiki becomes both a tool for choice and action and a flexible and extensible environment for collaborative negotiation of meaning' and so follows the Vygotskian approach (Carr et al. 2007). However, the successful employment of wikis to catalyse the move from 'knowledge representation' to 'knowledge construction', argues Morley (2012), requires careful guidance or scaffolding by the tutor. Inadequate tutor involvement, support or explanation of how the wiki collaboration would work was commonly cited as a reason for difficulty (Cole 2009;Greenhow, Robelia, and Hughes 2009;May, Burgard, and Abbasi 2011;Morley 2012). Other problems noted with wiki use by students include being 'locked out' of editing when another student is using a page which can result in a loss of work (Duffy and Bruns 2006); frustration with the change in study pattern from last-minute essay writing, to sustained, lengthy participation (Minocha Research (Morley 2012); 'unproductive interactions' and 'threats to privacy' (Greenhow, Robelia, and Hughes 2009, p. 251); and discomfort in the assessment process (Carr et al. 2007).
The current study sought to test the value of wiki use using an independent and highly regarded dependent measure Á final exam performance. Whilst we might suppose, in line with the previous literature, that wiki use benefits collaboration, writing skills and comprehension of course material, this study allows us to assess the important 'bottom line': Are the benefits of wiki use substantial enough to have a measurable effect on student assessment? Although previous research has shown some evidence for a difference in final grades between those who do and do not use a course wiki (Laru, Näykki, and Järvelä 2012;Ravid, Kalman, and Rafaeli 2008) this research has not addressed the confound that more engaged and enthusiastic students are more likely to both use a course wiki more and to score well on the final exam. The present study is the first, to our knowledge, which does attempt to disentangle these factors.

Methods
Participants in the investigation were level 2 students (typically in their second year of a 3 year course studying for a BA or BSc in psychology) at the University of Sheffield, UK. The majority of undergraduate students in our department are female (83%) and UK nationals (90%). Ninety-three percent identify as White British. The investigation considered their use of a wiki and final exam grade as part of a cognitive psychology module, PSY241, which forms a core part of the undergraduate degree. As a core module the content is regulated by the British Psychological Society. Topics covered include memory, attention, reasoning, language and vision. Teaching focusses around an introductory cognitive psychology textbook (Andrade and May 2004), with 22 one-hour lectures and three practical classes taking place over 12 weeks. The final exam consisted of three essays from a choice of six, completed over 3 hours. It was graded on a 100-point scale, as are all other exams at the University. Over 2 years, we collected data from students who took the exam.
We used 'media wiki' (http://www.mediawiki.org) to create a course wiki. Only registered users could edit the wiki, a device which prevented vandalism and allowed us to track use of the wiki. The wiki was introduced in course lectures and initial exercises and optional on-going tasks were set to encourage engagement with the wiki. These exercises included writing content on the course topics, leaving questions for other students and proofreading. However, at no point was use of the wiki monitored during the course, nor was it implied to the students that use of the wiki was being monitored or was a part of the course assessment. Wiki use on the course was relatively unstructured and undirected by the course leader, with minimal formal integration of the wiki into the course, it was merely provided for student use. In other words, we left it open to individual students to decide on whether to use the wiki without hope of approval or fear of recrimination. This means that it is sadly not possible to give full details of how the wiki was used by each student. Although students were assigned the task of editing the wiki, and directed to focus on writing about the course material, they were not told what to write or how to organise the wiki beyond a skeleton structure based on the principle topics of the course. Students in the second year of the investigation were told that use of the wiki, according to the evidence available from the first year, was an effective study strategy.
The analysis was approved by the University of Sheffield, Department of Psychology Ethics Sub-Committee (DESC), and carried out in accordance with the University and British Psychological Society (BPS) ethics guidelines. The use and exam performance data was collected incidentally and so did not require any change in the teaching or behaviour of the students involved. For this reason, the institutional review board waived the need for written informed consent from the participants.

Results
Data were collected from 216 students who took the examination over a 2-year period. The average grade for the students who took part in the study was 55.6 (standard deviation 10.5). For each student, it was also possible to collect their exam grades for the other modules they took in their second year of study (in the same semester and the semester after the cognitive psychology module). The average score on other level 2 psychology modules was 61.1 (standard deviation 7.23). This was roughly in line with previous years' results.
We assessed wiki use by automatic logging of number of edits, the number of different articles edited and the number of different days on which the wiki was edited. A simple correlation between number of edits and final exam score was significant (Pearson's r00.287, pB0.0001). The correlation between number of edits and average score on other second year modules was also significant (Pearson's r 00.589, pB0.0001). A regression, predicting PSY241 grade from number of wiki edits and average of other level 2 exam scores was significant (see Table 1) [R 2 00.367, F(2, 202) 058.633, pB0.0001].
In contrast, we also collected self-reported ratings for the first cohort of the study on seminar attendance and perceived usefulness of study seminars, along with reports of wiki use and perceived usefulness, and these had no significant predictive value for exam scores [R 2 00.071, F(4, 35) 00.592, p00.671].
A moderation analysis was conducted to test for an interaction between wiki edits and average level 2 grade. The overall regression remained significant [R 2 00.368, F(3, 201) 038.975, pB0.0001], as did the contributions of the number of wiki edits and average second year grade, but the interaction between these two factors was not significant (weight 00.009, b00.034, t00.0390, p 00.697). This suggests a consistent effect of wiki use on PSY241 grade across the range of average exam performance.
To illustrate these effects, we used a median split to divide the students according to their typical grade on other psychology courses, producing a 'low' and a 'high' group. We also categorised students according to their use of the wiki, non-user (0 edits, n068), light users (1Á3 edits, n 074) and engaged users (4 or more edits, n074). The results can be seen in Figure 1.
Statistically, the beneficial effect of wiki use can be understood as the equivalent of engaged wiki use predicting a 4Á5 percentage point gain on final exam score, which 5 moves a student, for example, from a mid 2.1 grade to a first class grade in the standard British undergraduate degree classification system. Analysis using the other indicators of wiki use Á number of articles edited and number of days on which the wiki was edited Á yielded similar results to the analysis of number of wiki edits. Additionally, a confirmatory factor analysis showed that a latent variable, composed of all the wiki use indicators, also remained an independent and additional contribution to PSY241 grade.

Discussion
As expected, there is a correlation between students' PSY241 exam grades and the average grade on other courses. Students who work harder and/or have greater academic skill will tend to get good grades on most courses. Those who work less hard generally and/or find academic work more difficult will tend to score worse on most courses. Although this study is a field study rather than an experimental design, the regression allows us to control for any general tendency to get good grades, and so to look specifically at the additional benefit of using the wiki. This is shown to be reliably and strongly associated with better exam performance. For both students who tended to get good grades, and those who tended to get less good grades, using the wiki provided a benefit in terms of final grade. Engaging with the wiki, defined as a mere four or more edits, was enough to gain students, on average, an extra 4Á5 percentage points on the final exam. The results are all the more convincing because the self-paced, undirected act of writing for the wiki is very different from the timepressured, question-focussed act of writing for the final exam. Our suspicion is that wiki use showed its benefit by acting on the most important hidden variable in pedagogy, comprehension.
Without doubt the quality of contributions to the wiki would also predict benefit, but we did not assess this, focussing on the raw count of number of edits made. As well as circumventing the tricky issue of how to assess quality of contribution, the focus on number of edits as an index of contribution keeps the emphasis of this investigation on wiki use as a beneficial process, rather focussing on the contents of the wiki edits. We choose to analyse wiki use from this 'zoomed-out' perspective, acknowledging that if wikis are found to benefit exam scores there is substantial work remaining to be done to characterise exactly how wikis are used and why they have any benefits.
That the current research is a field study, rather than an experimental design means we are able to take advantage of the high ecological validity of using real students engaged in a real learning task, and to use a real measure of learning (the final course exam). The disadvantages include that we have no experimental control and so cannot confidently ascribe causality to wiki use. We attempt to account statistically for hidden variables such as talent or enthusiasm which might create correlation between wiki use and exam performance, something which is novel in the existing literature on the benefits of wiki use in educational settings. Nonetheless, the ultimate test of the idea that wiki use benefits learning would be to run an experiment in which students were randomly assigned to either a wiki use condition or a control condition which was assigned a non-wiki task. Future investigations would also allow more detailed measuring of the nature of students' engagement with the wiki so as to test exactly what aspects of wiki use lead to the exam performance benefits. As it is, the focus of our investigation means we can only develop informed speculation as to the likely cognitive mechanisms by which wiki use enhances learning. We now present a review of plausible mechanisms of action for wiki use on learning.

The cognitive benefits of wiki editing
The study of human memory has produced a number of findings which can help us understand why wikis might be effective when used for study. A key idea is that of 'levels of processing' (Craik and Lockhart 1972). Studies such as those of Craik and Tulving (1975) showed that more elaborate processing of study items, e.g. thinking about their meaning rather than their physical form, enhanced subsequent memory for those items. Wikis encourage deep semantic processing of information, as the studentÁeditor is forced to consider how their addition to the wiki fits with the wider context of the extant material. An advance of the levels of processing account is that of 'transfer appropriate processing' (Morris, Bransford, and Franks 1977), which attempts to understand memory effects in terms of the overlap between encoding and retrieval processes. For subjects which are assessed by written work, wiki editing has an obvious advantage over textbook or lecture note study in terms of this framework also. Comprehension is assessed by producing written material, so, from this perspective, learning which involves writing has an advantage over learning which merely involves reading.
Editing a wiki also makes the course material personally relevant, another factor which has been shown to enhance memory for study items (Rogers, Kuiper, and Kirker 1977). Self-generated material has also shown to be easier to remember (Slamecka and Graf 1978), and writing an essay or editing a wiki is an obvious way of turning teacher-generated material into self-generated material.
Memory investigators have also noted the importance of mental schema for comprehension and memory (Bartlett 1932). Experiments have shown that students who are able to integrate novel information into an existing schema will remember it better (Stein and Bransford 1979), the so-called 'Assimilation Principle'. Wiki editing ensures that students engage with the organising schemas of the material. Exciting recent evidence has confirmed that practice at retrieving information in memory is a more effective memory strategy than rehearsing that information (Karpicke and Blunt 2011). Writing is a kind of retrieval practice, since relevant material must be continuously recalled and considered.
One factor which has been shown not to improve recall is intention to learn (Postman 1964). In light of this, wikis can play a role in encouraging more useful study habits than reviewing textbooks or lecture notes, activities which, once comprehension is achieved, are probably only lightly linked to subsequent recall of material.
Of course, there is far more to successful study than successful memorisation. Outside of the area of memory research, we also suspect that use of a wiki can facilitate comprehension and creative thinking about course material, although a solid body of research findings to support this suspicion is currently lacking. Making mistakes is linked to higher performance in sporting domains (Ericsson 2006). The process of editing and re-editing a wiki provides a natural, socially driven, arena in which to make mistakes about the interpretation of material, and to discover that they are mistakes.
Learning by writing 'Students learn to write by writing' is a truism and common starting point for scholarly consideration of teaching academic writing (Anson et al. 1993). Use of the wiki encourages students to communicate as both readers and as writers, developing the internal model of a scholarly audience that a writer needs. The wiki encourages intellectual autonomy because students must decide both what to write and how to phrase it. The wiki also encourages critical awareness, both because of this need for decision making about writing and because of the self-reflection promoted by committing ideas to writing in a public forum. Reviewing peers' writing has demonstrable benefits for students' own writing (Cho and MacArthur 2011).
The use of a wiki as a collaborative writing tool, whilst it does simulate collaborative writing and peer-review activities which are key to profession academic writing, also raises its own issues, not least of which is to problematise issues of plagiarism (Coffin et al. 2003).
One of the main challenges to successful collaboration through the use of wikis appears to be the mismatch between the explicit instruction to work together, and the implicit, but well-established understanding of grade-competition between peers: '[a]ssessing collaborative activities within an institutional culture of individual achievement sends mixed messages about what is valued' Á and so students were often anxious to have their own contributions seen and assessed separately from the whole (Naismith, Lee, and Pilkington 2011, p. 241). This is something which was, informally, reported by the students in our study. Hemmi, Bayne, and Land (2009) consider the benefits of web 2.0 technology in higher education to be severely limited by the current academic climate of individual achievement. Concerns about tracking individual contributions, copying, or worries about unfair individual assessment were found in a number of other studies also (Carr et al. 2007;Minocha and Thomas 2007). Moreover, there is a risk that students can be confused about whether they are being assessed on the subject matter or the web-page (Clougherty and Wells 2008).
Surprisingly, perhaps, students in some case-studies have also not found the wiki experience one which helped them feel more closely connected to their fellow students: 'interesting but a bit lonely' commented one student (Hemmi, Bayne, and Land 2009). Students in another study felt socially disconnected from other wiki users, and that T. Stafford et al. the lack of non-verbal means of communication was a source of 'misunderstanding and potential conflict' (Minocha and Thomas 2007, p. 204). However, in the literature as a whole, the potential benefits of wikis for student collaboration and group work are well evidenced, and the drawbacks possibly more to do with inadequate instruction and a preoccupation with individual achievement. The present work suggests there is potential for wikis to augment learning even within a traditional exam based course, and in spite of limitations and challenges identified by previous research (May, Burgard, and Abbasi 2011;Naismith, Lee, and Pilkington 2011;Wheeler, Yeomans, and Wheeler 2008).
Wikis are just one of a cluster of social-web technologies used in education which include blogs, twitter and Facebook (Manca and Ranieri 2013;Tess 2013). Wikis put a particular emphasis on the collaborative writing and rewriting of material. For this reason, we believe that wikis bear investigation in their own right.

Conclusion
The potential for the use of wikis in education has not gone unnoticed (Hazari, North, and Moreland, 2009;Konieczny 2007) but this study provides direct evidence of the concrete benefits of wiki use to an acknowledged indicator of pedagogic outcome Á namely final exam grades. We provide an example of how wiki use can be integrated into an existing course, and with minimal oversight and little emphasis on explicit collaboration can enhance both the student experience and learning outcomes. Additionally, the use of wiki technology provides students with a skill with wide application beyond that of the original course and programme of study. Future work should focus on both the pedagogic conditions which allow the benefits of wikis and isolate the cognitive mechanisms of action.