Worth it ? Findings from a study of how academics assess students ’ Web 2 . 0 activities

School of Medicine and Department of Information Systems, The University of Melbourne, Melbourne, Victoria, Australia; Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia; School of Computer Science and IT, RMIT University, Melbourne, Victoria, Australia; School of Business Information Technology, RMIT University, Melbourne, Victoria, Australia; Faculty of Information Technology, Monash University, Melbourne, Victoria, Australia; School of Languages and Linguistics, The University of Melbourne, Melbourne, Victoria, Australia


Introduction
When introducing a new assignment to university students, academics routinely encounter the question, 'What's it worth?' as students calculate how much of their overall result for the subject is at stake.As they gauge the benefits and risks of adopting a new method to improve teaching or assessment, academics themselves may also wonder whether it will be worth the effort.
This paper reports findings from a survey and interviews with Australian academics undertaking summative assessment of students' Web 2.0 activities.These data provide Moonen 2008, pp.100Á101).The visible and open nature of content and communication in Web 2.0 is at odds with an academic culture that has traditionally supported the making of decisions about assessment alone and in private (Hattie 2009).As yet, the academic policies of the authors' own universities, and possibly elsewhere, are silent on Web 2.0's assessment implications, such as the identification, ownership, safety, recording, privacy and preservation of student work.
Using Web 2.0 for assessment has significant implications for academic integrity, a term holding complex and contested meanings for academic authorship, textual ownership and the practices of acknowledging influential ideas and their media of expression (Thompson 2008).It challenges extant referencing and citation protocols to embrace a style of academic writing that captures the dynamic, dialogic and multi-authored nature of texts and artefacts generated in Web 2.0 (Gray et al. 2008).
The need to improve the practice of assessment in higher education is widely recognised (e.g.Boud 2000).Assessment's impact on student learning is well established (e.g.Rust 2002), including technology-based assessment (e.g.Kirkwood and Price 2008).If done well, assessment can support and reinforce the intended student learning; that is, assessment can work 'for' or 'as' learning (Earl 2003).Web 2.0 may well be able to offer affordances for every stage of assessment, from designing and implementing it, through supporting and grading it, to evaluating its effectiveness (Waycott et al. 2010a).Web 2.0 might offer ways to make assessment more authentic, personalised, engaging or problem-oriented (Elliott 2008).
There is a growing body of case-by-case accounts evaluating various approaches to assessing Web 2.0 activities (e.g.Farmer, Yue, and Brooks 2008, Ng and Hussain 2009, Lai and Ng 2010).Whitelock (2010) analyses eight case studies, and Waycott and Sheard (2011) present six more, illustrating Web 2.0 uses in assessment 'for' learning, assessment of learning processes, and overcoming discontinuity between formative and summative assessment.However, to date, there has been no systematic investigation into the aspects that academics find worthwhile Á or conversely, not worth it Á about assessment of student Web 2.0 activities.
The prospect of opportunities for improving assessment but equally significant challenges, gave rise to a national Australian study in 2009.This study aimed to paint a detailed picture of where, why and how academics were using Web 2.0 for summative, as well as formative, assessment of student learning.It aimed to fill gaps in our knowledge and provide a foundation for developing a more widely shared understanding of the pedagogical worth and assessable value of students' Web 2.0 activities in higher education.

Method
Academics were invited to participate in an anonymous online survey and an optional semi-structured interview during the period August to October 2009.Human research ethics approval for this project was granted by each of the partner universities.We advertised the survey in national learning and teaching newsletters, inviting participation from any academic who had set assignments where their students used Web 2.0 technologies to do work for which they earned subject marks.We also issued invitations to academics who had recently published papers about their use of Web 2.0 tools in higher education.Web 2.0 technologies were defined as any of the following: blogging and microblogging, audio and video podcasting, social bookmarking, social networking, wiki writing or virtual worlds; using any tools, such as Twitter, Flickr, WordPress, Second Life, YouTube, Delicious, Facebook, or tools bundled in Learning Management Systems (e.g.Blackboard's wiki tool).
Survey respondents answered questions about one assignment: one graded task that had been required in one subject or unit of study.A total of 44 questions covered the way the assignment was designed and implemented, the marking process, student results, policy and procedure considerations, and background information about the subject.Multiple responses were possible to many of the questions.The survey included three open-ended questions about the assignment.A copy of the survey instrument and detailed reporting of these data are available at http://web2 assessmentresources.wikispaces.com/4.'Further'Resources.
Semi-structured telephone interviews conducted by one researcher explored interviewees' assessment practices and their perspectives on the use of Web 2.0 for assessment An interview protocol (Appendix 1) was used, although the precise questions varied from case to case, allowing conversation to develop organically.
Sixty-four academics responded to the survey, of whom 53 completed all or most questions, and 22 were interviewed.Most survey respondents (81%) identified their role as 'subject coordinator', for example, having overall responsibility for learning, teaching and assessment in the unit of study.Many respondents (37%) had first used the Web 2.0 assignment they described in 2009; 48% had begun the assignment between 2006 and 2008.All open-ended survey responses and interviews were transcribed and analysed first separately and then together by four of the researchers, using open coding.Care was taken to account for both opportunities and challenges, i.e. positive and negative reports of experiences.Interview and survey findings were then categorised by two of the researchers using the organising principle of the different stages in the assessment cycle.
We could have categorised these data in many ways, following the literature; however, many of these categories would have failed to provide insights into summative assessment practices as well as into the assessment process as a whole.Franklin and van Harmelen (2007) have proposed recommendations for writing assessments involving Web 2.0; the majority of these are formative assessment, and Hatzipanagos and Warburton (2009) too make a case for exploring the links between formative assessment and social software.Bartlett-Bragg (2008) has proposed a framework for designing Web 2.0 assessment which highlights how much effort is needed in the initial planning stages, while Bower (2008), Bower, Hedberg, and Kuswara (2010), and Conole et al. (2004) have published methods for matching learning tasks and assignments with learning technologies which also emphasise the degree of preparatory effort required.To describe the experience of working with Web 2.0 assessment more fully than this, we chose to analyse the data using the organising principle of the different stages in the assessment cycle: not only designing the assignment, but also implementing and supporting the assignment with students, marking the assignment and giving summative feedback to students, and quality management of the assessment processes (Rust, O'Donovan, andPrice 2005, Bloxham andBoyd 2007).

Findings
Descriptive statistics and comments from interviews and open-ended survey responses illustrate key benefits and challenges that academics weighed up at each stage of the assessment cycle.

Designing the assignment
Findings about the design of Web 2.0 assignments give a mixed picture of the extent to which academics are attributing value to such approaches.The 62 responses to a question about the type of Web 2.0 tool used showed that wiki writing (52%) and blogging (50%) were most commonly used.Uses of social networking (27%), audio or video podcasting (26%), virtual world activities (19%) and social bookmarking (18%) for student assignments were reported less often.
Web 2.0 assignments were designed for a range of settings, including all year levels in 35 undergraduate degrees and 16 postgraduate coursework degrees, and in both small and large classes.While 23% of respondents were working with classes of 30 students or fewer, 25% had classes of between 31 and 100 students, and 25% had more than 100 students (the largest class reported had over 500 students).
Differences were apparent in the value of these assignments for different fields of study.In almost half the 50 responses to the 'field of study' question, the assignment was part of a course in Humanities or Social Sciences (42%).Almost a third of assignments came from each of Information Technology (32%) and Education (30%).Mentioned less often were Health and Medicine (16%), Management and Commerce (12%), Creative Arts, and Natural and Physical Sciences (6% each), Architecture, Engineering, and Law (2% each).
Web 2.0 assignments appeared equally useful for assessing various types of learning outcomes: generic or graduate skills or attributes (55%), specialised knowledge or skills required in a discipline or profession (44%) and foundation knowledge or skills preparatory to a discipline or profession (45%).These assignments were valued not only for demonstrating outcomes but also for facilitating learning processes, for instance: . Optimising peer review: 'They get their [peer] feedback on the assignment and . . .improve it based on the feedback' (Interviewee 6). .Extending the learning community: 'It really reaches out their communication beyond the boundaries of the classroom . . .it's not just my view anymore; they can go and look at other people's views as well' (Interviewee 8). .Encouraging ongoing engagement: 'to make sure that [students] did something continuously through the semester'.(Interviewee 20) Interviewees cited a number of challenges in getting the assessment design right.Some of these were common to any assignment, for instance the challenge of making room for student creativity while also providing sufficient structure: Other challenges arose from needing to learn by trial and error how a Web 2.0 assignment would work: We didn't understand how the interaction would unfold, how the tool would behave, a lot of downtime and template restrictions.(Interviewee 18)

Implementing the assignment
Findings about how academics are introducing and supporting Web 2.0 assignments with their students suggest that it is not a simple matter to do this well.Compared to more conventional assignments, the extra efforts required to get students to work effectively with Web 2.0 may detract from the perceived worth of the activity.
Students commenced work on most assignments either at the start of the study period (55%) or in the middle of the study period (34%).The time they were expected to spend on the assignment was most often 11Á20 hours (33%), followed by 1Á10 hours (27%), 21Á30 hours (17%) and 31Á40 hours (11%).Students were allowed more than a month to complete most assignments (66%) or at least several weeks in a number of other cases (22%).
Students were expected to do the assignment using their own IT facilities (88%) or IT facilities provided by their university (78%).They were expected to work on it in various locations including: . On campus in class (25%). .On campus but out of class (39%).
Academics interviewed spoke at length about their need to develop new strategies to help students interact online: 'I would give them an example at the beginning in the first week or second week on how this would work and the sorts of comments that I would be expecting'.(Interviewee 15) Several interviewees described the work involved in encouraging students to draw on their personal experience to generate an online identity: 'There's a process that goes into them finding their different voices, how to share appropriately, how to write with authority'.(Interviewee 8) Academics reported coaching students in co-creation and collaboration: 'I said it had to be [500 words] that you had to add and then you had to value-add to the content.So we talked about what we meant by that'.(Interviewee 22) Providing adequate support for students posed new workload challenges for some staff: [There is a lot of] work involved in setting up [the blog assignment] and making sure all the students know how to do it.If you ask them to write an essay they just go off and write it, you don't have to spend the first three weeks of the course teaching them about essays.(Interviewee 11) It's hard yakka [work] for the teachers . . .because we often have to be on [online] almost all hours at times . . .You tend to do that to give the support.(Interviewee 1)

Marking and giving feedback on the assignment
Findings about marking and giving feedback on Web 2.0 assignments indicate that at this stage of the assessment cycle most academics are not taking up the open and collaborative approaches to feedback and assessment that the social web affords, but are exercising their individual authority and discretion as assessors.
The value of Web 2.0 assignments as a proportion of a student's overall mark in the unit of study ranged from very little to 100%, with most assignments worth up to 50%, as shown in Table 1.Most students received marks plus other feedback, including confirmatory comments (67%), explanatory comments (64%), and corrective comments (61%).
The majority of Web 2.0 assignments were marked by one staff member (63%), with just over a quarter marked by more than one (27%).Academics gave various responses to a free-text question asking them to identify strategies they used to brief other markers, including students who had to mark each other's work, as shown in Table 2. Peer-assessment or self-assessment was factored into marking in only 14% and 11% of cases respectively.
Establishing a framework for evaluating the work that students produced was not straightforward.One interviewee found it difficult to set criteria that were appropriate to different levels of study, noting that the differences were 'sort of formally in place with essays, you know, the number of words you write as a Masters student is different to a Grad Dip but when it comes to creating [Web 2.0 content], perhaps it's a bit less clear' (Interviewee 7).In some assignments the diversity of content types that students could produce created another challenge for marking: '[Students] are producing very different things, so I suppose the criteria need to capture that in some way' (Survey response).Another survey respondent noted that the different style of writing in a Web 2.0 assignment made marking difficult: 'because the content does not have to be of an academic standard it can be hard (to) assess, e.g.marking sloppy or lazy prose'.
In some instances calculating a final mark for individuals' work was complicated by the need to assess how it had been influenced by class members' running online exchange of feedback and interaction.Twenty-three percent of respondents said that equal marks were allocated to everyone in a group project.
Interviewees reported that they had developed new marking techniques to respond to Web 2.0 affordances in a variety of ways: We have a complete history of everything they do . . .So we can actually look in and see what this particular group has excelled at, which one gives us a wide variety of results so that we can separate the good students from the others.(Interviewee 13) The assessor is not assessing a written document, they're assessing a page which . . . is a whole labyrinth of choices and connections, so they've got to actually work their way through . . .decision paths and decision tree and things like that.(Interviewee 10) In particular, there were many instances where staff found merit in being able to combine formative and summative assessment, for example: I leave comments on the blogs as we go through but usually engaging with the content, not sort of marking it, but then I can add those sorts of comments against the rubric when I assess them.(Interviewee 11) Sometimes assessing process as well as product was problematic, for instance when marking co-created wiki content: How do you mark assignments when students can change/overwrite each other's work!Many students who contributed early, found that their work was completely lost.How do you manage this process of overwriting and still contributing to the same content?(Survey response) Some academics marked students' work in stages (34%).However the multilateral peer-to-peer interactions and repeated student-to-staff interactions during a cumulative Web 2.0 assignment could put pressure on staff responsible for marking large numbers of students:

Managing the quality of assessment processes
In terms of overall assessment management, using Web 2.0 for student assignments holds clear value for some academics but a level of risk for others.The survey addressed academic administration and quality management by asking participants to respond to a series of statements about their assignment in relation to major assessment policy issues.These were distilled from a review of Australian universities' Table 2. Measures taken to prepare assessors of student Web 2.0 assignments.

Steps taken and illustrative quotes Number of responses
Criteria/criterion referenced/graded marking sheet: 'Criteria and marking sheets agreed before term start, adhered to by all involved staff who meet to agree detail'.

8
Rubric: 'Rubric provided to all prior to commencement of these Units.Discussion with students as they commence marking'.

7
Experienced lecturers/tutors do the marking: 'Only experienced markers mark these assessments and they liaise before marking to ensure criteria are commonly understood'. 4 Meeting: 'Meeting held between the tutors and course coordinators to standardise marking'. 3 Peer review/panel review: 'Students engage in online peer review of their draft write ups on [a] peer review system'. 2 Examples provided: 'The lecturer provides their own answers to the questions posed, and discusses them with the markers before they mark the e-journals'.
2 Inspection: 'Throughout the marking period the markers keep an eye on all the students' blogs and comments made by other staff'.
1 publicly available assessment policy and procedure documents.Table 3 shows that most academics were confident that they were managing their assignment appropriately in most respects.However there were noteworthy levels of non-alignment or uncertainty in matters such as offering supplementary assessment, keeping students' marked work on file, safeguarding students' identity and privacy online, and recording students' learning in ways that would allow it to be tendered as evidence of meeting prerequisites for further study (e.g. for recognition of prior learning).Several interviewees described how Web 2.0 affordances supported the quality management of assessment in terms of monitoring progress, reinforcing academic integrity, and protecting students' moral rights.For example: Students start off being able to see their own posting.So they can't copy off one another.And then later on they see the other postings but then they can't edit their own.So it truly is their own work because that first bit is done in their creative sense.(Interviewee 12) One of the reasons why I chose to go with the blogs and make it an informal writing exercise (was) because then I could get a sense of their voice, get a sense of consistency as well . . .so when it comes to marking the formal assessment tasks that's another plagiarism check.(Interviewee 21) Academics also noted key challenges for quality management of assessment, such as 'being able to guarantee ongoing access to materials hosted on sites outside the university' (Survey response) and 'all those issues of confidentiality and copyright' (Interviewee 20).As Interviewee 13 stressed: I tell the students over and over again that it is on the World Wide Web, it's not associated with the university, be careful what you put up there, make sure you are comfortable with this.

Discussion
These findings show that academics have found many features that are worthwhile, and many ways to see value in designing, conducting, marking and managing student Web 2.0 assignments.They report a range of challenges and a few real risks: some familiar to anyone changing their assessment practices but others unique to the assessment of student Web 2.0 activities.
It is noteworthy that a relatively small number of academics responded to the survey and many of them had used the Web 2.0 assignment for the first time that year.We may speculate that other academics did not respond either because their use of this form of assessment was less well developed, or because they did not use this form of assessment at all.Although the use of student Web 2.0 activity in university education is rising perceptibly, our findings show that low and medium-stakes assignments are most common.This is consistent with our previous research, which indicated that Web 2.0 assignments are offered chiefly for optional enrichment or as preparation for a more conventional written assignment or examination (Gray et al. 2010).
Academics in our study, the majority of whom were from Humanities and Social Sciences disciplines, were clearly finding value in the opportunities that Web 2.0 afforded for assessment of openly published and collaborative student work.As these academics would appear to represent the prime essay-writing disciplines, it could be proposed that written text remains the most assessable of the many Web 2.0 forms of activity.According with this, and consistent with the types of innovation most commonly reported in the literature, we found that Web 2.0 assignments most  frequently took the form of blogging and wiki writing.The academics still identified challenges with these affordances of open publishing and collaborative writing.Even though they were willing to put effort into coaching students in the use of these new forms, they remained concerned, for instance, that students' text-building efforts were not 'academic' enough.Academic staff seemed to be trying to deal with these issues as isolated practitioners, innovating through trial and error.Interestingly, most respondents were subject coordinators, potentially responsible for teams of tutors and markers, yet they volunteered little evidence of open or collaborative marking practices, and there was little suggestion of novel methods used to evaluate the effectiveness of their assessment practices.There would seem to be substantial room for more creative uses of social media technologies in assessment to automate or streamline marking; to devise new forms of feedback; and to incorporate multilateral assessment, making use of peers or even extending Whitelock's (2010, p. 322) concept of a 'community' of assessors beyond the classroom.The use of Web 2.0 technologies may prove too disruptive for conventional assessment practices, shaking the foundations of two hitherto solid pillars of contemporary assessment thinking.The division between formative and summative assessment and the straightforward alignment of assessment tasks with objectives may ultimately be re-engineered by Web 2.0 assessment forms.In the Web 2.0 environment, formative and summative assessment can appear like a double helix Á and perhaps become indistinguishable.As the process of content creation is transparent for the duration of the task and after, this makes it possible (and arguably raises student expectations) that summative assessment will attach value to the processes that students use to create their assessable products.It may be necessary to revisit constructive alignment between the learning activity required and the learning outcome intended (Collis and Moonen 2008).

Research in Learning Technology
Even when academics undertake solely summative assessment in these assignments, the affordances of Web 2.0 mean that some formative feedback is most likely inbuilt.Thus, the work of arriving at a student's final mark can Á and arguably must Á explain how the student's response to feedback has added value and accrued marks along the way to the later summative grade.A third pillar of assessment convention therefore may require re-engineering here: As well as the students' content creation being made highly visible, the often very private evaluative work of giving formative and summative feedback and the detail of the actual feedback given can become more transparent than ever before, not only for a student cohort, but also for academic colleagues and others.This represents a shift from academics conducting assessment in a 'black box' (Hattie 2009).
Thus, in using Web 2.0 for assessing student learning, questions about which (and whose) 'achievement' is to be assessed may, at least in the short term, become more contested.Is it the student who has been prepared to take individual risks (which may not, in the end, have been totally successful), or the student who has collaborated wholeheartedly with their peers?Is it the student who has advanced in their progress towards a professional identity, or the student who has shown particular talent in using Web 2.0 tools?And what of the academics?Is it the one who has introduced a specific Web 2.0 assessment form for learning a particular technique or concept, or the one who has redesigned assignments to be more engaging for students?Is it the academic who has found a method to facilitate staged assessment, or the one who has devised a method for monitoring group work?An academic without a sound rationale for assessing students' Web 2.0 activities will struggle to justify the added effort flowing from the assessment design Á effort which may involve addressing critical questions about the allocation of extra attention and support for the duration of the task; about the calculation of results for students' learning outcomes; and about the propriety of interactions between university students and others who are active in the wider social Web.
A limitation of this study is that it offers an overview of academics' self-reported practices only.We cannot be certain whether there were others who chose not to respond and what their reasons might have been.Still needed are in-depth comparative studies of individual academics' decisions in using Web 2.0 for assessment, as well as a fuller understanding of university students' perspectives on these kinds of assessments.Accordingly, a follow-on study from the survey reported here involves semester-long observation and document analysis in 17 diverse subject learning and teaching case studies of Web 2.0 assessment, as well as interviews and focus groups with participating staff and students (Waycott et al. 2010b).

Conclusion
Most advocates of using student Web 2.0 activities in higher education offer little detailed advice on how to conduct assessment which comes to grips with the unique features of Web 2.0, its difference from previous forms of student writing and staff marking or its academic administration.This study raises important considerations for teaching academics, for educational developers and technologists and for learning and teaching leaders and policy-makers.For individuals or groups who have heard enthusiastic commentaries about the assessment potential of Web 2.0.our research offers new insights, derived from broadly based empirical data, into the chequered realities of current practice.
It may be inadvisable for intending innovators to follow the example of our study participants in proceeding conservatively and applying Web 2.0 affordances selectively to suit their pre-existing pedagogical context.Nor does is seem to be a good idea to work on these pedagogical innovations as a sole practitioner of Web 2.0 assessment in one's academic department or discipline.In terms of educational efficiency and effectiveness, incremental and individualistic uptake of Web 2.0 may not achieve sufficient scope and scale to deliver substantial improvement of academic assessment, and so may not be worth it in the face of workload and policy concerns.
Institutionally and collegially concerted shifts in this practice have more likelihood of adding the anticipated value to higher education.So far, although reform of student assessment in higher education is called for Á 'The revolution will encompass 'feedback from assessment' and the development of visible learning and visible teaching' (Hattie 2009) Á and despite the potential of Web 2.0 technologies to facilitate such change, Web 2.0 cannot be seen to be transforming university assessment.On the contrary, its affordances are being appropriated, in some cases, to perpetuate conventional, invisible university assessment practices (even to the extent of constraining it inside online Learning Management Systems).This trend is at odds with the view of some commentators that its very openness is what makes the social web most valuable.Staff and students who are ideologically inclined towards this openness are increasingly likely to seek to disrupt the status quo of assessment standards and policy with 'challenges concerning collaboration, evaluation, and the general aim and status of the material produced by students' (Dohn 2009).
For those whose role is to encourage and support innovation in higher education, it is clear that the majority of teaching staff and students too will need broadly-based, well-informed advice and support to make the most of the assessment potential of Web 2.0.There is much work to be done before we can be confident that using Web 2.0 for assessment is safe, fair, engaging and worthwhile for students and staff.
Citation: Research in Learning Technology 2012; 20: 16153 -DOI: 10.3402/rlt.v20i0.16153 The bottom third of the class had difficulty thinking about what to post on [i.e.blog about] when it was left completely up to them . . .If you give too much guidance are you constraining the best students?If you don't give enough guidance you're leaving the weaker students to flounder.(Interviewee 11)

Table 1 .
Web 2.0 assignment as a proportion of students' overall mark.

Table 3 .
Reported alignment with academic policies and procedures in assessment of student Web 2.0 activities.