Collaborative learning with PeerWise

Denis Dureta*; Rob Christleyb; Paul Dennyc; Avril Seniora

Correspondence: *. Corresponding author. Email: E-mail:


Abstract

Building effective and supportive communities of practice in an asynchronous environment can enable students to learn from each other at their own convenient times without the need to meeting for discussing concepts. In an undergraduate course with a large amount of content to learn, working collaboratively to answer practice exam questions can help encourage deeper learning and understanding. This project examined whether exam performance can be positively influenced by online collaboration during revision and suggests that the quality of an individual’s contribution has a direct effect on his/her subsequent exam performance. We compared two student cohorts’ exam marks from their previous 3 years of exams with how they performed in Year 4 after using PeerWise (an online repository of multiple-choice questions that are created, answered, rated and discussed by students). In addition, we looked at the quantity and quality of their contribution in PeerWise to evaluate how this affected their fourth year exam performance. We confirmed that exam performance is improved by authoring questions in PeerWise, but found it is further enhanced by replying to, and commenting on, others’ posts as well.

Received: 2017 June 21; Accepted: 2017 November 15

RLT. 2018 Jan 3; 26: 10.25304/rlt.v26.1979
doi: 10.25304/rlt.v26.1979

Copyright

@ 2017 D. Duret et al.

Keywords: Keywords word, peer-assisted learning, collaborative learning, computer-supported collaborative learning.

Introduction

Face-to-face peer-assisted learning has been successfully implemented for clinical skills learning with various cohorts, including veterinary students (Baillie et al. 2009; Reid et al. 2017) and medical students (Bennett, O’Flynn, and Kelly 2015; Field et al. 2007). Furthermore, effectiveness in higher education has been acknowledged (Topping 1996).

The development of online peer-assisted learning is more difficult, as there are no face-to-face meetings to discuss issues. Typical characteristics of online systems, including asynchronous communication and anonymity, present both advantages and challenges.

The undergraduate veterinary science course at the University of Liverpool is an intensive 5-year programme. The volume of course material is relatively high, which means that the students are often strategic in their learning, striving to achieve higher grades, focused on assessment demands, while managing their time and study effectively (Chigerwe, Ilkiw, and Boudreaux 2011). Referring to a study investigating students’ participation patterns in accessing and contributing to online discussions (Taylor 2002), those students could be described mainly as either ‘workers’, participating actively in the discussions, or ‘lurkers’, participating occasionally. The third group described by Taylor, the ‘shirkers’, is practically non-existent in the later years of the veterinary course.

During their fourth year, students are required to attend lectures from October to March followed by three summative examinations for each of the three disciplines (Small Animal Studies, Livestock Health and Welfare Studies and Equine Studies). These examinations are in the form of multiple-choice questions (MCQs) and, in order to help students informally with their revision, PeerWise1 was introduced in September 2013. PeerWise is a free web-based learning tool where students create, share, answer and rate MCQs. This tool had already been used successfully at other veterinary schools in the United Kingdom (Edinburgh and Glasgow). Furthermore, previous studies (Rhind and Pettigrew 2012; Sykes, Denny, and Nicolson 2011) indicated that authoring and answering questions has the potential to promote deep learning by establishing that positive correlations exist between these activities and subsequent exam performance.

PeerWise provides a platform for collaborative learning because students interact with their peers to generate a bank of practice questions. Our aim, in introducing this environment, was to support learners in becoming effectively engaged in all aspects of the course, rather than simply revising superficially with limited understanding of the content and the context of what they were learning. Online collaborative learning has been defined as ‘a learning process where two or more people work together to create meaning, explore a topic, or improve skills’ (Harasim 1995). Furthermore, it has been suggested that the use of the term ‘collaborative learning’ rather than ‘peer teaching’ is more appropriate when there is no cognitive distance and the teaching task is not explicit but informal (Ten Cate and Durning 2007). This description applies to the current study because the students engaged in the process belonged to the same cohort, and the usage of PeerWise was neither explicit nor formal. Collaborative learning has been often compared with cooperative learning, as the principle is the same; two or more students working together may learn more than an individual student working alone – ‘two heads are better than one’ (Bruffee 1995). However, Dillenbourg (1999) differentiates between these two: “In cooperation, partners split the work, solve sub-tasks individually and then assemble the partial results into the final output. In collaboration, partners do the work ‘together’.” Hence, considering that the goal for the students is to generate a large bank of practice questions, PeerWise provides a collaborative environment to achieve this.

The term ‘computer-supported collaborative learning’ (CSCL) was introduced by Koschmann (1996). This style relies primarily on active collaboration where the ‘learning takes place through interactions among students’ (Stahl, Koschmann, and Suthers 2006). Students are involved in the process of research and finding the answers to study questions together in a way that promotes self-directed leaning. The key to a successful CSCL environment is collaboration, which is a challenging task to achieve and requires careful planning and structured support. Student interaction is not always stimulating or constructive. CSCL can generate feelings of frustration (Capdeferro and Romero 2012); therefore, it is essential to be aware of the following pitfalls in social interaction (Kreijns, Kirschner, and Jochems 2003):

  1. Taking social interaction for granted.
  2. Restricting social interaction to cognitive processes.

The PeerWise learning environment offers the option of posting comments on questions created by individual students which can, in turn, engage other students in an online discussion. In effect, learning takes place through interactions. Students have the option to create mini-discussion boards that focus on a question, where they can seek or give further explanations, depending on their knowledge and expertise.

The aim of this study was to analyse those online contributions and consider the impact on performance in the end of Year (Y) 4 exams.

Methods

Data collection and analysis

Data for this study were obtained from three sources – two to assess the impact of PeerWise on performance and a third to assess students’ experience of using PeerWise.

To assess the impact of PeerWise on performance, data were extracted from two databases. First, the examination marks for each student were extracted from the University of Liverpool student record system. Two marks associated with each student were used for this study: The average mark obtained in the fourth year (Y4 mark) and the average mark for the first 3 years of the course (Y1–Y3 mark). Second, the students’ online activities using PeerWise were extracted from the PeerWise server, hosted by the University of Auckland, New Zealand. The two datasets were then merged and anonymised. The data were collected over a period of 2 years and hence included two successive student cohorts (Table 1). In addition, students’ contributions with PeerWise were retrieved, including the total number of answers submitted per student over a period of 3 months prior to the exams and the date and time of each contribution.

Table 1.. Dataset.
Number of students Number of students using PeerWise Number of questions created Number of questions answered Number of comments posted
Cohort 2013–2014 130 102 758 35 259 135
Cohort 2014–2015 144 109 1028 54 193 376
Total 274 211 2814 89 452 535

The online contributions of each student were analysed both quantitatively and qualitatively. For the quantitative analysis, students were divided into three groups:

  • Group 1 (PeerWise, PW1) – Never used PeerWise (63 students); that is, students who did not log in to PeerWise.
  • Group 2 (PW2) – Used PeerWise (143 students); that is, students who logged in to PeerWise but did not post any comments.
  • Group 3 (PW3) – Used PeerWise and posted comments (68 students).

Multivariable linear regression analysis was used to explore the relationship between the outcome, the average mark obtained in the fourth year (Y4 mark) and the explanatory variables, which included the average mark for the first 3 years of the course (Y1 –Y3 mark) and the use of PeerWise, which was categorised into three groups, as described above. For the analysis, the Y1–Y3 mark was centred, by subtracting the overall mean from each student’s score in order that the intercept in the regression model represents a meaningful value, that is, the mean Year 4 mark for students with the mean marks for Y1–Y3 and who did not use PeerWise (Group 1). The fit of the data to the key assumptions of linear regression were explored by plotting the observed data against the model residuals. All statistical analyses were conducted using the R Language for Statistical Computing (R Core team 2016).

Thematic analysis was used to identify, analyse and report patterns (themes) occurring in the mini-discussion boards. This was carried out by the first author (D.D.), as described by Braun and Clarke (2006). To further explore the effect of the length of contributions, PW3 was divided (68 students) into two subgroups: PW3a (low contribution) and PW3b (high contribution). The criteria used to divide PW3 were based on the total number of characters that each student contributed as comments and replies. A total of 34 students contributed less than 500 characters and were allocated to Group PW3a; the other 34 students were allocated to Group PW3b.

In order to assess students’ experience of using PeerWise, the students from the cohort 2014–2015 were also asked to complete a questionnaire (Table 2). The questions were designed to gain a broader understanding of the value of online collaboration as perceived by the students. Survey Monkey,2 an internet survey tool, was used to distribute the questionnaire, which was sent out to 144 students in April 2015, of which 70 students responded.

Table 2.. Questionnaire.
  1. (1)Did you use PeerWise to revise for the fourth online summative exams (Yes/No)?
  1. (2)If YES, could you describe how it helped you (or not)?
  1. (3)If NO, could you describe why you didn’t want to use PeerWise?
  1. (4)What would be the best feature that PeerWise has to offer?
  1. (5)What would be the worst feature that PeerWise has to offer?
  1. (6)Did you use the comments option in PeerWise to query a particular question or to reply to somebody else’s query (Yes/No)?
  1. (7)If so, could you describe how/if the asynchronous interaction was useful for your revision?
  1. (8)Do you have any other comments about PeerWise?

Results

Quantitative analysis

The linear regression model indicated that a student with the mean Y1–Y3 mark (67.84%) and who did not use PeerWise would be expected to have an averageY4 mark of 68.58% (Table 3). Use of PeerWise was associated with an increase in averageY4 mark. Most notably, the highest use category (PW3) which was interpreted to indicate contribution to discussions was associated, on average, with a 3.5% increase in averageY4 mark, compared with students in the reference category (PW1, i.e. did not use PeerWise). This effect was significant (p < 0.001) and the 95% confidence interval suggested that the true effect could be as small as a 1.1% mark increase to as much as a 4.3% mark increase.

Table 3.. Statistical analysis.
Univariable models Multivariable model
Coefficient Coefficient Standard error Lower 95% confidence interval Upper 95% confidence interval p
Model 1 Model 2
Intercept 68.58 66.4 67.08 <0.001
Centred Av Y1Y2Y3 0.95 - 0.94 0.04 0.86 1.02 <0.001
PeerWise -
PW1 - Reference Reference <0.001
PW2 - 2.44 1.61 0.70 0.24 2.99
PW3 - 3.50 2.67 0.82 1.06 4.27

Examination of a plot of the fitted values versus the residuals suggested that the assumptions of a linear relationship and of homoscedasticity were reasonable (data not shown).

To examine the relationship between the quantity of comments generated by students and their course performance, the total length of all comments posted (i.e. number of characters) was compared with the residuals/change in mark for each student in the PW3 group (Figure 1). Increased contribution was significantly associated with a higher than predicted mark: For every 10-fold increase in the number of characters contributed, the predicted mark increased by 2.1 (95% confidence interval: 0.4–3.8, p = 0.02).


[Figure ID: f0001] Figure 1.. Relationship between students’ contribution (number of characters) and deviation from predicted Year 4 marks (based on average performance in years 1–3) among students that commented using PeerWise (Group PW3). Dotted red line illustrates the regression line for the relationship between the log10 (number of characters) and the change in marks from the expected results. The solid red indicates the expected deviation (i.e. 0) from the predicted Year 4 mark, if PeerWise had no effect.

Qualitative analysis

The qualitative analysis focused on the comments from both cohorts of students. Nvivo3 10 was used to code each student’s comment with the five themes that had emerged (Table 4).

Table 4. Themes used for qualitative analysis.
Description Example
Irrelevant Low-level contribution I got excited by knowing that A was correct. I didn’t bother reading on/let’s hope we aren’t this tired for the real thing!
Courtesy Showing appreciation Ah okay maybe a change in the wording of the question would be useful :) thanks.
Insightful Seeking more detailed explanation Would you really arthrodese a stifle?
Rephrasing Summarising the discussion Yes and the cranial cruciate ligament does not contribute anything to the stability of the patella; therefore, this option is not appropriate!
Further work and analysis High-level contribution Indications for stifle arthrodesis:
  • Animals with unreconstructable joint fractures
  • Stifle luxation
  • Severe OA
  • Grade 4 patella luxations which have not responded to medical management
(From Atlas of Orthopedic Surgical Procedures of the Dog and Cat)

Figure 2 shows an example of interaction between students, starting with an insightful comment, followed by rephrasing, then further work and finally courtesy acknowledgement.


[Figure ID: f0002] Figure 2.. Comments associated with question 1 371 400, course ID 9235 (Cohort 2014–2015).

To further explore the trend identified in Figure 1, a qualitative analysis of the comments posted on PeerWise was conducted, focusing on the PW3 subgroups (PW3a and PW3b). The method used to subdivide PW3, based on the total number of characters, produced a comparable division of the students to that obtained when division was based on the total number of comments (results not shown). In Group PW3a, 50% of the students only posted one comment and 90% of the students had posted four-or-less comments with the remaining 10% posting five-or-more comments. In Group PW3b, 90% of the students had posted five-or-more comments, with an average of 14 comments per student. Comparison of PW3a and PW3b (Figure 3 and Table 5) indicated that PW3b (‘heavy users’ of PeerWise) scored, on average, 3.7% more in their fourth year exams.

Table 5.. Mean and number of comments for groups PW1, PW2, PW3a and PW3b.
Number of students Mean Y1–Y3 mark (%) Mean average Y4 mark (%) Minimum/maximum number of comments posted Average number of comments posted
PW1 63 67.2 66.4 0 0
PW2 143 68 68.9 0 0
PW3a 34 69.1 67 1/8 2.3
PW3b 34 69.2 70.7 3/61 13.2


[Figure ID: f0003] Figure 3.. Students’ performance for groups PW3a (students who contributed less than 500 characters) and PW3b (students who contributed at least 500 characters).

Figure 3 also illustrates that both groups (PW3a and PW3b) included students with a range of previous marks, evenly distributed from average marks in years 1–3 below 60% to above 80% (i.e. fair to good students being represented in those groups) and that the course performance of students in Group PW3b was better than those in Group PW3a.

Nvivo case nodes were created for each student with attribute values as PW3a or PW3b, corresponding to the five subgroups defined earlier (Table 4), with their number of characters posted. The comments associated with each student were extracted from PeerWise as two sub-datasets:

  1. ‘Top-level comments’: 275 comments written as feedback to answered questions.
  2. ‘Replies to comments’: 236 additional explanations comments written (i.e. appended to existing top-level comments).

The datasets were coded separately and each written input from the students was coded according to the themes defined in Table 4. The results are presented in Table 6.

Table 6.. Number of comments made for each of the themes.
Irrelevant Courtesy Insightful Rephrasing Further work and analysis
Top-level comments PW3a 3 (7.5%) 2 (5%) 23 (57.5%) 11 (27.5%) 1 (2.5%)
PW3b 25 (10.5%) 11 (5%) 102 (43%) 86 (36.5%) 11 (5%)
Reply to comments PW3a 6 (24%) 5 (20%) 3 (12%) 11 (44%) 0 (0%)
PW3b 15 (7%) 75 (35.5%) 12 (6%) 91 (43%) 18 (8.5%)
Total PW3a 9 (14%) 7 (11%) 26 (40%) 22 (34%) 1 (1%)
PW3b 40 (9%) 86 (19%) 114 (25.5%) 177 (40%) 29 (6.5%)

The students who appeared to benefit the most from PeerWise were those contributing the most in terms of the number of characters posted (Figure 1). Among this group, and considering the top-level comments category (Table 6), students from Group PW3a posted 20% less comments in the Rephrasing theme than in the Insightful theme. For the Group PW3b, the difference between those themes was considerably smaller (6.5%). This might indicate that students from Group PW3a were more interested in getting answers to their problems than responding to somebody else’s problem. In the reply to comments category, nearly a quarter (24%) of the posts from Group PW3a were in the bottom theme (Irrelevant theme), compared with just 7% for Group PW3b. Finally, only 1% of the students from Group PW3a posted a comment in the top theme (Further work and analysis theme) compared with 6.5% from Group PW3b.

Questionnaire

The questionnaire responses (from the cohort 2014–2015) indicated that 57 students (81%) found PeerWise useful and 38 students (59%) posted comments either to query a question or reply to somebody’s else query.

Students reported that PeerWise helped them with the required revision, as intended:

It was very helpful being able to practice answering MCQ questions and having this tool available online.

It was useful as a bank of questions to test myself after I had done a section of revision. It broke up the monotony of revision. Allowed me to appraise gaps in my knowledge.

The fact that PeerWise can be used asynchronously was reported to help students to create mini-discussion boards focusing on individual questions:

Very helpful to be able to have group discussions.

Good because if an answer that was most popular was different to that suggested by the author, then it could be discussed.

I found it was useful as a mini-discussion board so people could share thoughts about the question.

The anonymity was also an important aspect for students joining the conversation:

The anonymity really helped to gain courage to query something.

However, this mode of revision did not suit everybody:

It wasn’t very helpful and confused and stressed me.

The quality of the social interaction and the questions was raised by a few students:

Some people got petty which didn’t help but some of them were really helpful to clarify.

Some of the questions are really rubbish and should be moderated.

Almost half (n = 70, 49%) of the students responded to the questionnaire, with 12 students (8%) self-identifying as not having used PeerWise. The reasons for not using PeerWise varied from time constraints to different revision techniques:

Didn’t have time to use it as well as revise from my notes.

Prefer to use my own notes.

Participation

Student participation on PeerWise, per day, revealed increasing activity during the weeks immediately preceding the examination days (Figure 4).


[Figure ID: f0004] Figure 4.. Number of answers submitted per day over a period of 3 months for 2014–2015 cohort (examination days highlighted in red).

Discussion

The results of this study indicate that students who used and contributed to PeerWise (i.e. PW3) benefited the most from PeerWise with respect to the improvement in their course performance in Year 4; these students scored on average 3.5% more than predicted if they had not used PeerWise. A likely explanation for this is that students who contributed more to PeerWise were more fully engaged with the concept of collaboration supported by this online platform. However, those who engaged with PeerWise might have also participated in other forms of learning (i.e. reading about related concepts in a textbook) before contributing.

Overall, this suggests that using PeerWise only to author and answer questions, and to ‘observe’ or superficially comment on others’ questions, limits the learning opportunity. Students in this category could be likened to ‘lurkers’ on social media and other online platforms, who simply access the resources but do not actively contribute to the collaborative process. Taylor (2002) found out that the academic performance of the ‘lurkers’ was on average not much less than that of the workers. However, the veterinary profession encourages its members to be proactive with their peers and other professions. Therefore, institutional support and facilitation to engage more fully and collaborate using PeerWise may be needed to help those with a tendency to lurk.

Participation timeframe

The timeframe over which students participate may also have a bearing on the usefulness of the activity. Some students spend the last few days, or even hours, before an examination urgently revising (i.e. cramming). Figure 4 highlights clearly this phenomenon, with the number of answers submitted per day rising dramatically in the 2 weeks prior to the examinations. This is not the best way to use PeerWise as it is an asynchronous system. In most cases, there is not an instant answer to a question. Social interaction using PeerWise normally takes place over a period of 3 days (Figure 2) where students work and learn together to answer a question. Typically, queries are answered within a few days and then the activity on the mini-discussion board stops. Therefore, a student posting a query about a question just before the exams might not receive a timely reply. The usefulness of the activity would be greatly improved with a more consistent commitment from the students throughout their revision.

Within a collaborative environment, the level of commitment and the quality of individual contribution vary (Capdeferro and Romero 2012). For this study, 211 students used PeerWise but less than a third of them (60 students) posted a comment either seeking further explanation or replying to a query. A recent study (Bates et al. 2014) found that 75% of the questions, created as part of the summative assessment in introductory physics courses, can be classified as high quality. It could be considered therefore that two-thirds of the students did not participate in the mini-discussion boards because the high quality of a number of questions did not require further discussion.

Anonymity

With PeerWise, students were able to post comments anonymously which enabled them to participate in a non-threatening environment. It is an essential part of PeerWise, and it was identified to build confidence and competence alongside the ability to ‘lurk and learn’ (Hooper, Park, and Gerondis 2011). They were able to fully engage and develop their knowledge by asking and answering questions without revealing their identity. Anonymity is a recurrent theme demonstrated by another study in the same institution: ‘Students have unnecessary concerns about displaying an answer, even for a formative assessment’ (Duret and Senior 2015). Students should be discouraged from a reliance on anonymity as the veterinary profession requires its members to be prepared to discuss their decision, which can be a ‘best guess’ answer in some real-life situations. Confidence needs to be fostered in veterinary undergraduates, which happens during clinical rotations where members of the staff are working with small groups of students (maximum of five students) or in team-based activities (maximum of eight students). This fostering cannot be replicated with PeerWise as members of staff are not included in the collaborative learning.

Encouraging students to engage more with PeerWise

The benefits of collaborative learning were only realised when students actively engaged in the online collaborative process. Further encouragement may be beneficial to students who would otherwise not fully engage; however, the effect of such encouragement on these students needs further evaluation. PeerWise was introduced to fourth year veterinary students as an aid to revision and some students chose not to participate until the last few days of the activity. Ideally, this engagement should happen weeks if not months before the exams so that students have time to interact socially and construct their knowledge.

Conclusion

PeerWise has already been used at other veterinary schools in the United Kingdom showing its potential to promote deep learning. Our study makes an additional contribution by demonstrating that the improvement in exam performance is greater for those students engaging with the questions through posting or replying to comments.

In this study, we have not been able to determine whether engaged students were predisposed to use PeerWise and so were already more likely to do better in the exams, or if PeerWise alone increased the engagement. Further work should help to clarify this.

Our results do suggest that students who are less confident in their performance and knowledge need to be supported to use PeerWise. Facilitation may be the key here, by introducing PeerWise in earlier years. For example, a student could be expected to contribute a question and be given a ‘credit’ for doing so. A staff member could constructively critique the input and give feedback. Such an approach may have a positive impact in later years, where engagement may be improved as students have familiarity with the activity and have been ‘trained’ in how best to leverage this ‘community of practice’.


Notes
1..

en1PeerWise was created by Paul Denny (Department of Computer Science, The University of Auckland).

2..

en2SurveyMonkey is an online survey development cloud-based software (www.surveymonkey.com)

3..

en3NVivo qualitative data analysis Software; QSR International Pty Ltd. Version 10, 2012.

References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.

Refbacks

  • There are currently no refbacks.