Contrary to assumptions, the use of SRSs such as clickers is not new; they have been in use since at least the 1960s (Judson and Sawada 2002). There is a large body of research already on the use of SRSs in educational settings, which has been reviewed extensively (Caldwell 2007; Fies and Marshall 2006; Judson and Sawada 2002; Kay and LeSage 2009; Simpson and Oliver 2007). The vast majority of this research concurs that using SRSs in the classroom has a positive or at least benign influence, especially on factors such as student engagement, participation and peer learning potential (Caldwell 2007). A common thread through nearly all of the literature is the importance of combining SRSs with social constructivist pedagogies: ‘it is instructional method itself that is important when considering implementing an electronic response system’ (Judson and Sawada 2002, p. 178). The critical task is to deploy SRSs to ‘provide the tools for creating authentic learning environments and fostering the communication channels that support the social construction of learning and understanding’ (Massingham and Herrington 2006, p. 98).

One of the more thorough literature reviews, by Kay and LeSage (2009), identifies areas for further research. They report that further research needs to be undertaken in a broader range of discipline areas, as most of the implementation and research so far has been conducted in mathematics and hard sciences, a finding that has been echoed elsewhere (Judson and Sawada 2002; Simpson and Oliver 2007). This is now changing: Habel (2009) explored the use of SRSs in academic skills lectures for international students; and SRSs have been used in Marketing (Sprague and Dahl 2010), Management (Keough 2012) and Nursing (Patterson, Kilpatrick, and Woebkenberg 2010). This is about more than the challenge of applying an established pedagogical technology to a new context, though: non-scientific disciplines are more likely to contend with divergent rather than convergent questions, which is a challenge for technology and e-assessment more broadly (Crisp 2007). A related question which is largely unexplored by the literature is the type of students who find the most benefit from using SRSs; Sprague and Dahl (2010) discovered than both international students and students with low need for cognition particularly enjoyed its use: ‘This finding is important because it validates this technology as a valuable methodology to facilitate learning with students who typically do not enjoy more traditional methods inherent in a university learning environment’ (p. 97).

Furthermore, while Kay and LeSage (2009) indicate that several authors recommend that SRSs be used in conjunction with explicit student preparation for lectures, these pieces do not go much beyond this recommendation. D'Inverno, Davis and White (2003) do recommend the provision of lecture notes before (rather than after) lectures, but the outcomes and effects of providing skeletal notes prior to Engineering Mathematics classes are reported in only the most anecdotal fashion. In addition, much research into the use of SRSs seems to focus on specific pedagogical strategies in relatively isolated contexts. There is a strong consensus that ‘clickers are not, by themselves, sufficient to increase subjective reports of engagement’ (Morling et al. 2008, p. 49). According to Kay and LeSage (2009), three of the most important factors – engagement, peer-based instruction and student preparation – were hardly explored in conjunction (with the exception that three authors focussed on engagement and peer-based instruction). Finally, Kay and LeSage note that there are no accounts of why students find SRSs so engaging and recommend the collection of qualitative data to help explore this factor.

Our action research design developed directly from the gaps in the literature identified above. To begin with, our work is in a different context to most work in SRSs: although they are being used more in human sciences and business disciplines, there are to our knowledge no applications of their use in humanities or law-related fields. It is particularly important to explore the use of SRSs in the context of divergent questions where the emphasis is on interpretation, discussion and the use of evidence to support claims. This responds to ‘the need for a concerted research effort, one that rigorously explores conditions of use across diverse settings and pedagogies’ (Fies and Marshall 2006, p. 101). More specifically, the use of SRSs in this discursive context of divergent questioning supports the constructivist pedagogies that are essential to the implementation of SRSs.

Furthermore, SRSs were used explicitly in conjunction with both attendance and preparation activities. It is not yet common for pedagogical strategies to explicitly address preparation for large lectures (for an exception see Michaelsen and Sweet 2008), and this research both emphasised preparation to students and built it into the questions asked via the SRS. This was combined with an investigation into attendance, engagement and peer-based instruction to explore how these factors combined to support student learning in large lectures. Finally, qualitative data from surveys were drawn on to try and elicit reasons for enhanced engagement through SRSs and to attempt to explore the type of students for whom SRSs are particularly useful.

The research questions we sought to address were as follows:

Can the use of SRSs in a large first-year law lecture promote student engagement?

Use of a cyclical action research methodology allowed us to interrogate our second research question, which is directed to the complex question of which combination of techniques is most effective to achieve our desired goal of increased student engagement. This project built on two separate implementations which created cycles of research with increasing levels of rigour. Habel (2009) used VotApedia with international students to enhance engagement with issues around academic integrity, plagiarism and the use of text-matching software. Although higher engagement was casually observed, it was not properly documented or analysed. In 2010, the second author began a preliminary implementation of VotApedia in Principles of Public Law. Again, heightened engagement was observed and responses in end-of-semester evaluations were positive, but this was not sufficient to properly establish the effect of the intervention. Therefore, the current study was designed to explicitly explore the effect of SRS usage on peer-learning, student preparation and engagement through the use of VotApedia in large first-year law lectures, a discipline that has not previously been used for such implementations.

The implementation was inspired by the work of Eric Mazur, who pioneered peer-learning pedagogies through using SRSs (Lasry, Mazur, and Watkins 2008). In each of the 3 weeks of our project, students were provided with a two-page pre-reading material and asked to complete that reading before attending the lecture. As in all weeks of the course, students also had advance access to the lecture slides, a topic guide and set readings which were designed to introduce them to the topic being examined, and which were built into the SRS activity.

In the lecture, we asked three substantive questions arising from the assigned pre-reading. For each question, students were first given a short lecture introduction to the topic (5–10 minutes), before being split into groups (of approximately three students) to discuss which answer was correct.2 One student from each group would then vote (using their mobile telephone) for their group's preferred answer from a choice of four given responses (see Figure 1). Once voting was completed, an interactive discussion would follow, during which student volunteers explained the reasoning behind their choices to the whole class, the results of the voting were revealed and discussed, and the strengths and weaknesses of each potential answer were examined in a group discussion led by the lecturer. We did not seek merely a ‘correct’ answer, but engaged in the processes of higher-level reasoning leading to a more complex understanding of the strengths and weaknesses of different approaches, thus pursuing the divergent exploration of issues necessitated by the research design. This process would then be repeated for the remaining two substantive questions.3

In addition to the substantive questions posed, we also asked some questions whose purpose was purely to assess the benefits of our use of mobile phone voting to engage students through explicitly requiring preparation. Thus, at the beginning of each lecture, students were asked if they had completed the pre-reading, and how confident (on a 1–5 Likert scale) they were about their ability to correctly answer the questions posed. At the end of each lecture, students were asked if they found the pre-reading useful to their understanding of the issues addressed. The project implementation and research design is illustrated in Appendix.

This process was conducted in three 2-hour lectures during the course (in weeks 8, 10 and 11 of the semester). In addition, we undertook observational analysis of the learning environment during our classes: the primary author was available solely for observation, while the second author was engaged in delivering the lecture and simultaneously observing. At the conclusion of the final session, students were asked whether (on a 1–5 Likert scale) the lectures with mobile phone voting were ‘more engaging than Matthew's regular lectures’ which, as explained above, regularly involved interactive question-based activities and group work.

After these lectures, we administered an optional on-line survey to elicit further information about students’ attitudes and perceptions of the use of mobile phone voting in lectures. This follows the approach of valuing self-report of engagement and satisfaction in the Australian Survey of Student Engagement by the Australian Council for Educational Research (ACER) (Coates 2008), as long as it is combined with other data. Students also provided feedback on mobile phone voting in their Student Experience of Learning and Teaching (SELT) surveys completed in the final class of the semester. It could be argued that these SELT results are overly contextual, in that they only focus on student experience in one specific course. However, because all these students have previous experience of different courses and lecturers, and the SELTs follow an institution-wide format, students do actually respond in the wider context of their experiences at university. This variety of data sources allowed for triangulation (as recommended by ACER) to effectively analyse the impact of VotApedia upon student preparation and engagement in a divergent questioning context.

In the first week of the project, we asked students to answer each question first individually, then to discuss their answer in small groups and vote again. For each of our three questions, as reported in Table 1, group discussion led to a measurable improvement in student performance.

These increases were statistically significant (p<0.05 for the first two questions and p<0.10 for the third),4 which substantiates the value of group discussions to student understanding, confirming the importance of constructivist pedagogies in the deployment of SRSs (Judson and Sawada 2002). This corroborates our suspicion that the benefits of peer-learning through SRSs that have been observed in the hard sciences also apply to more humanities-based disciplines, and should motivate educators to consider using SRSs in a broader range of disciplines.

The results for student preparation are presented in Table 2. Looking just at data for the first 2 weeks, there is encouraging evidence that students are prepared to put in the effort to do the pre-reading once they see the benefits that it can bring. Thus, in week 1 the pre-reading was done by 64% of students and valued by 70% of students. In week 2, this positive experience from week 1 prompted increased preparation – 74% of students did the pre-reading and 80% found it valuable. The negligible standard errors suggest that the results are valid.

This positive response in week 2 might generate an expectation of increased preparation in week 3. However, the extent of preparation in week 3 dropped to only 42%. In week 3, our lecture clashed with a major assessment in another course (in which most of our students were enrolled). This clash with assessment was a stronger (negative) influence than the (positive) influence of increased engagement.

In all cases, the standard error of the proportion was within acceptable limits, suggesting that students’ self-reports of completion and usefulness were valid. Given the informal and anonymous nature of the responses there is no reason to suspect that students would misrepresent their views.

The impact of the preparation of assessment items on student attendance has been previously noted (Corbin, Burns and Chrzanowski 2010). What this research adds is the impact of assessment items, even in other courses, on engagement – in the critical week where there was a clash between our lecture and assessment completion for another course, the rate of reading even our short two-page pre-reading was noticeably lower. This emphasises the importance of thorough planning across courses and even programs, where possible, or perhaps a stronger focus on time management or the management of expectations with students (Figure 2).

A number of different data sources suggest that mobile phone voting led to increased student engagement in our course. Observational analysis from both authors indicated much more energy and liveliness in the room during mobile phone voting, and particularly sharp spikes when ‘correct’ answers were revealed. In addition, at the end of week 3 students were asked whether they found lectures with mobile phone voting ‘more engaging than Matthew's standard lectures’. There was 84% broad agreement5 that they were, as reported in bold in Table 3.

This data is supported by the evidence of student perception gained from responses to SELT surveys at the end of the course. Responses to the prompt ‘This person encourages student participation’ (on a 1–7 Likert scale) are reported in Table 4, comparing the 2011 results with those obtained in 2010, when the same course was taught by the same staff member.

Comparing the 2 years, there is a marked increase (8%, p=0.033 on a two-tailed t-test) in student perceptions of the lecturer's encouragement of participation. The introduction of mobile phone voting was the only substantial intentional difference in course methodology between the 2 years, as previous years employed interactive, social constructivist pedagogies involving explicit preparation for lectures and group work and discussion.6 Indeed, in the previous year the lecturer even used a paper-based quiz to encourage engagement through questioning and so the inclusion of technology was part of an on-going pedagogical development. In this longer-term context there was still improvement via the use of the SRS. Furthermore, this was the only SELT measure of student experience to change dramatically between the 2 years, which suggests that it was mobile phone voting that caused this increase in student perception of the encouragement of participation by the lecturer.

At the end of the semester an online survey was administered to elicit students’ own perceptions of their engagement through the use of mobile phone voting. This survey went well beyond a simple evaluation of the teaching and learning experience: it was explicitly designed to explore relationships between attendance at lectures where VotApedia was used and subjective reports of engagement, using both quantitative and qualitative data. This relationship between attendance and engagement is an under-researched area. Students were asked how many VotApedia lectures they attended; since there were only three the margin of error is small and it can be assumed that their responses to this question were accurate. Based on their response, students were then asked how engaging they found Matthew's ‘normal’ lectures and how engaging they found the VotApedia lectures. Results are reported in Table 5.

These results indicate significantly higher self-reports of engagement based on the use of VotApedia (p<0.05 for students who attended two or three VotApedia lectures; although the result for the small number of students who attended only one lecture was not statistically significant), but they must also be interpreted with caution. There is clearly some selection bias operating here; obviously students who are already engaged in learning are more likely to both attend more lectures (including VotApedia lectures) and to complete the survey. Here we encounter the common problem of eliciting results regarding students who are disengaged. Furthermore, it is clear that many students already feel that lectures are engaging, so the use of VotApedia is building on a base of strong pedagogical practice. Additionally, there were pockets of dissatisfaction with the use of VotApedia that must not be discounted, and appeared more strongly in the qualitative data. Nonetheless, these results are encouraging and demonstrate a clear link between attendance and engagement when SRSs are used to supplement an already interactive lecture style.

In addition to Likert-style questions, students were asked open-ended questions which provide more nuanced information. These comments were also grouped according to attendance at VotApedia lectures to further explore relationships between attendance and engagement and were subjected to a thematic content analysis based on an initial interpretation of the comments. The themes that emerged from the data included positive, negative and ambivalent assessments of VotApedia; comments that VotApedia made the lecture interactive, engaging or interesting; suggestions that it should be used more in lectures; and comments that it used too much time that would be better spent otherwise. To ensure complete representation of the responses, some responses were double coded: for example, if a statement had a generally positive comment followed by a comment that it took too much time, it was included in both categories (Table 6).

Clearly, the overwhelming qualitative response was that students felt that the use of VotApedia was engaging. For example, ‘I'm a student who often finds myself in ‘la la land’ in lectures, no matter how interesting the topic/lecturer, so for me it really helped to make me focus and actually learn something!’ This indicates that SRSs are particularly useful for students who are usually disengaged. Many respondents used adjectives such as engaging, interesting, or interactive, while others referred to group discussion or had more sophisticated ways of expressing a deeper relationship with the material or learning context. Many students had generally positive comments about the use of VotApedia, and some even suggested that the technology helped them retain information for quizzes and the exam, indicating that it may also serve traditional functions of helping memory retention. For example, ‘I have hence noticed a significant benefit in the use of VotApedia in regards to my exam revision—I have retained more information from these lectures in comparison to the ‘normal’ ones’. Several of these positive comments revolved around how VotApedia motivated preparation; for example, one responded that they only found it useful ‘after doing the pre-readings—I found out the hard way!’ This again reinforces the links between attendance and engagement, as facilitated by the use of SRSs.

Despite these overall positive responses, there was also notable ambivalence: several respondents combined positive comments (even about engagement) with criticisms of the time VotApedia took out of lectures. Others felt put upon (one student found it ‘Distracting and quite frankly, oppressive’), or that the material was being ‘dumbed down’, although often while simultaneously noting the positive aspects of VotApedia. A good example was the observation that: ‘Although the voting was ‘engaging’ and interesting, I felt that it took far too much time that would have been better spent simply giving students more information’. This sort of concern is indicative of some deeper student ambivalence about the purpose of lectures, which is becoming more widespread with increased use of technology in general. More particularly, this might be the reverse of a previous finding: that students who actively dislike SRSs are those who traditionally succeed in a didactic lecturing environment.

Indeed, this comment about inappropriate use of time was quite common. A total of 27 students (19% of survey respondents) commented that VotApedia took too much time and recommended fewer instances of it, or lamented the reduction of content or depth in the lectures. Nonetheless, our analysis demonstrates clearly that the vast majority of students (often including those who were dissatisfied with the use of time) found our use of mobile phone voting to be very appealing. Moreover, as we have already observed, part of the time taken was in asking questions necessary for our research (including whether the pre-reading had been done and whether it was useful). We are confident that some of the concerns about the time involved in obtaining and discussing the audience response would be addressed in a 2-hour lecture involving only three substantive questions.