Kjetil L. Nielsen*, Gabrielle Hansen, and John B. Stav
Faculty of Technology, Department of General Science, Sør-Trøndelag University College (HiST), Trondheim, Norway
(Received 18 June 2012; final version received 8 October 2012; Published 11 June 2013)
Abstract
In this article, we describe and discuss the most significant teacher-centric aspects of student response systems (SRS) that we have found to negatively affect students’ experience of using SRS in lecture settings. By doing so, we hope to increase teachers’ awareness of how they use SRS and how seemingly trivial choices or aspects when using SRS can have a significant negative impact on students’ experiences, especially when these aspects are often repeated. We cover areas such as consistency when using SRS, time usage, preparation, the experience level of the teachers with regard to SRS, teacher commitment and attitudes, teacher explanations, and how students fear that voting results can mislead the teacher. The data are based on 3 years of experience in developing and using an online SRS in classroom lectures, and they consist of focused (semistructured) student group interviews, student surveys and personal observations.
Keywords: audience response systems; clickers; student attitudes; teaching pitfalls
*Corresponding author. Email: kjetil.l.nielsen@hist.no
RLT 2013. © 2013 K.L. Nielsen et al. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons “Attribution 3.0 Unported (CC BY 3.0)” license (http://creativecommons.org/licenses/by/3.0/) permitting use, reuse, distribution and transmission, and reproduction in any medium, provided the original work is properly cited.
Citation: Research in Learning Technology 2013, 21: 18989 - http://dx.doi.org/10.3402/rlt.v21i0.18989
A student response system (SRS) can be described as an electronic voting system that presents students with a multiple-choice question, often as part of a quiz, to which they will answer with a small handheld device (commonly referred to as a “clicker”), usually after engaging in peer discussions. Benefits from utilising such a system in lectures include increased student motivation and engagement (Draper and Brown 2004; Masikunas, Panayiotidis, and Burke 2007), easier clarification of misunderstandings (Rice and Bunz 2006), promotion of active learning (Boyle and Nicol 2003) and increased student performance (Hake 1998; Rao and DiCarlo 2000), including better conceptual understanding (Crouch and Mazur 2001).
With such a promising record, it is easy to forget that SRS is only a tool that, if not used correctly, can actually be detrimental to the lectures (Barber and Njus 2007). Focusing primarily on the technology with a belief that the technology will automatically improve lectures, instead of focusing on how students think and learn, is the single most important reason for failure when implementing new technology into education (Mayer 2005), and SRS is no exception.
There are several publications that give best practice guidelines for using SRS (e.g. Caldwell 2007) in classroom lectures. Most publications on SRS focus on its benefits, often with guidelines for increased learning and effective SRS use. Some publications also report on teaching aspects of SRS that have been shown to decrease students’ satisfaction with the system. These include inefficient use of time with SRS (Knight and Wood 2005; Reay, Li, and Bao 2008) by, for example, using too much time setting up the system and handing out clickers (Draper and Brown 2004; Masikunas, Panayiotidis, and Burke 2007) or having discussions drag on for too long (Dufrense et al. 1996); responses from SRS being graded (Knight and Wood 2005; Reay, Li, and Bao 2008); grading of SRS questions resulting in mandatory attendance (Graham et al. 2007); SRS being used only to keep attendance (University of Wisconsin 2012); SRS being used just for the sake of using it (Draper 2002); teachers having negative attitudes towards SRS (Draper 2002); irrelevant clicker questions (Barnett 2006); and lack of teacher explanation after the quiz (Smith et al. 2011).
Although this seems to be an elaborate list, these negative aspects are mostly briefly mentioned and not described in depth in the literature (for a list of general challenges with SRS, see Kay and LeSage 2009). In-depth studies on these aspects and how they affect students’ experience of SRS are, to the authors’ knowledge, lacking. To fully understand and appreciate best practice guides on SRS, it is important to understand how and why different aspects of implementation can have a negative impact on students.
In this article, we describe and discuss such aspects after 3 years of experience (since 2009) in developing and using an online SRS for modern handheld devices, such as smartphones, at Sør-Trøndelag University College in Norway. We start this article with background information, describing the classes where SRS was used and the implementation choices for the different years of testing. This is followed by a brief description of research methods and a presentation of our results. We conclude this article with a discussion and conclusions.
Preparatory engineering classes at Sør-Trøndelag University College consist of students who wish to start studies for an engineering degree but do not have the required subjects from senior high school. There are four classes, with approximately 50–70 students per class, and five different courses are taught: physics, mathematics, social science, Norwegian and English. One of the classes, however, has only physics and mathematics. The courses last for a full year, and the curriculum is close to that of the second and third years of senior high school, but some changes are made to reflect that the students are to begin work on an engineering degree. During the years of testing, the classes usually consisted of approximately 84–86% male students, with an average age of approximately 22–23 years.
The lecture format with SRS during the testing was the traditional teacher-style lectures (using digital blackboards) normally used in the courses, with the teacher presenting the students with 1–4 quizzes during the session. The SRS sequence could vary, for instance by using the peer instruction method (Mazur 1997), which consists of an individual thinking period, followed by a vote, group discussions and a revote. Other examples could be to exclude the first voting session and go straight into the discussion of the alternatives, or have the students discuss, but not show, the alternatives until students are ready to vote. In none of these 3 years of testing did we have SRS responses affect students’ grades; in other words, SRS participation neither did give extra credit nor did answering correctly on quizzes. Below, we give a brief overview of the implementation choices (technical choices are not included). See Table 1 for a summary of the implementation.
| Semester | Number of classes | Subjects | Duration | Data |
| 2009A | 1 | p | 4 weeks | 1 survey and 2 interviews |
| 2010S | 4* | p | 8 weeks | 1 survey and 7 interviews |
| 2010A–2011S | 2 | M, p and s | Whole semesters | 1 survey and 4 interviews |
| 2011A–2012S | 4 | M, p, s, n and e | Whole semesters | 1 survey |
In the autumn semester of 2009, SRS was used in one class in physics, which was taught by one of the authors (Kjetil L. Nielsen) for 4 weeks. In the spring semester of 2010, SRS was used for 8 weeks in all four physics classes. The 8 weeks of testing were part of a study to compare different SRS sequences (Nielsen, Hansen-Nygård, and Stav 2012). We prepared around 50 quizzes, which the teachers used for the duration of the study. All teachers used the same quizzes to minimise variables.
SRS was implemented in two classes and three courses (physics, mathematics and social studies) for the whole duration of the 2010–2011 semesters. Teachers were given no restrictions on the use of SRS in their courses, or on how much they wish to use it. Two teachers were assigned to make a pool of quizzes to be used in mathematics and physics. However, all teachers were free to ignore this pool and make their own questions. Before and during the semester, we had regular meetings and seminars with the teachers during which they were given the opportunity to use SRS, discuss methodological choices and so on. One of the authors (Gabrielle Hansen) acted as a contact person to whom the teachers could present questions about SRS or report bugs in the software.
In the last year of testing, 2011–2012, SRS was made available to all classes and teachers in the preparatory engineering courses. Seminars were held to discuss methodological choices with SRS before the start of the semester, but there were no SRS meetings during the semester due to time constraints among the researchers.
The data consist of student interviews and surveys from 3 years of SRS testing. In total, there were 13 focused (semistructured) group interviews consisting of around four students per group. The interviews were conducted by one of the authors (Gabrielle Hansen). During the first two years of testing, Gabrielle Hansen was present as an observer in the lectures where SRS was used. The observations (as well as feedback from teachers) would often function as a starting point for deeper investigation of students’ experiences (through the interviews). There were nine interviews from the 2009–2010 period, two of which were conducted during the autumn semester of 2009, and the rest during the spring semester of 2010. Four interviews were conducted at the end of the spring semester of 2011. Surveys were conducted during all 3 years of testing. Many of the questions from the surveys were constructed based on information that emerged from the interviews. Most survey questions or statements used a 1–5 Likert scale ranging from, for instance, “totally agree/very strong degree” to “totally disagree/very little degree.” Students answered all surveys anonymously.
The interviews from the different semesters were part of larger studies to investigate various aspects of SRS implementation. Students could sign up as volunteers for being interview subjects. Candidates from each class were chosen at random from this list (with the exception that we wanted at least one of each sex). Before the interviews, the students were given a presentation of their rights as interview subjects (such as anonymisation) and were informed of the general themes of the interview. The interviews started after the students had given their consent that they understood their rights and that the interview could proceed.
Interview guides were constructed to explore both the positive and negative experiences of students with SRS, and later interview guides were modified based on results from previous interviews and feedback from teachers. Even though the different studies had different foci, the interviews covered a broad spectrum of themes, including various aspects on how the teachers used SRS. The analytical tools consisted for the most part of a three-step coding and analysing scheme adapted from Charmaz (2001, 2003), which is a method for identifying experiences, feelings and attitudes. The results shown in this article are in large measure an accumulation of the most significant negative aspects that emerged from the interview analysis.
Our students experience SRS in general as a very positive element in their lectures and a valuable tool for both the teacher and students. In the 2011–2012 survey, 95.6% of the students answered that they would want SRS to be used in their future education if they had the chance (3.8% answered “don't know,” and 0.6% answered “no”; n = 160). Although the students are in general positive about SRS, there are several aspects that can negatively affect students’ experiences of the system. Some of these problems are due to technical difficulties, such as voting devices not being able to connect to the service. This was a noticeable problem during the first year of testing and a source of frustration among the students. As technical difficulties became negligible, and teachers could use SRS as they saw fit, most criticism from students shifted towards the way SRS was used by different teachers, especially when the students were in the position to compare different teachers.
The results are divided into six different sections: (1) consistency when using SRS; (2) differences in teachers’ SRS experience level; (3) time usage; (4) commitment: a two-way street; (5) preparation of the questions; and (6) voting results: a false image of understanding. The last section also contains two subsections on the importance that students place on the teacher's explanation and on including “don't know” as an alternative.
Before starting to use SRS in their lectures, teachers should have a clear goal of why they want to use SRS and be consistent in their choice. Inconsistency when using SRS gives students the impression that the teacher does not know why he or she wants to use it and more or less uses SRS for the sake of using it. The students value using SRS to receive instant feedback, give feedback to the teacher, or have a pause in the middle of the lectures where they can engage in peer discussions. Inconsistency, such as using SRS only occasionally, creates uncertainty among the students because they do not know how long they will have to wait for these benefits, and thus it creates a source of irritation and frustration. According to the students, inconsistency is also a sign of low teacher commitment (more on teacher commitment is given in this article). Students emphasise that if a teacher is going to use SRS, he or she should use it regularly or not at all. The following student quotes reflect why they did not value SRS in a specific course:
Student 1: Because it is so rarely used that.. it should have been more.. that he either used it more or not at all.
Student 2: I feel that it is used so little that there's like no point.
Student 3: Yes, I agree with you on that. I feel that I don't get much out of it [using SRS] in that course because it is used so little.
If SRS is rarely used, it will also feel less integrated as a natural part of the lectures. As well as being expressed in the interviews, this is also reflected in the surveys from the 2011–2012 classes (Table 2, question/statement 6 (QS6)), where the majority of students agree with this statement. Teachers who use SRS rarely will not receive enough practice using it, which in turn results in more fumbling with the software and in SRS use feeling less “smooth” to the students.
| Question or statement | Teacher experienced with SRS* | Teacher not experienced with SRS* | |
| 1. To what degree do you feel SRS is a natural and integrated part of the lecture? | 4.24 (0.77) | 3.34 (1.19) | |
| 2. I feel SRS compromises the lectures. | 1.62 (0.89) | 1.60 (0.84) | |
| 3. To what degree does SRS help you learn the subject matter? | 3.64 (0.90) | 3.19 (1.12) | |
| 4. Do you feel SRS is used too much or too little in the lectures? | 2.90 (0.52) | 2.39 (0.80) | |
| 5. How important is the commitment of the teacher with regard to your experience of SRS? | 4.21 (0.82) | ||
| 6. If SRS is rarely used, it will also feel little integrated with the lectures. | 3.62 (0.75) | ||
Differences in experience levels among the teachers are also apparent from the survey. Some questions in the survey were organised so that students could give different answers based on different courses. We separated the responses into groups consisting of teachers with previous experience with SRS and teachers who used SRS for the first time during the 2011–2012 period. Teachers with previous experience used SRS to a greater extent (QS4), and SRS was also regarded as a more natural part of their lectures (QS1). Students also felt that SRS helped them learn the curriculum better when it was used by experienced SRS teachers (QS3). A Mann–Whitney U test showed a statistically significant difference between teachers with and without previous SRS experience for QS1, QS3 and QS4 (p < 0.001 in all cases). There was no statistically significant difference (p > 0.05) between teachers’ experience levels with SRS in students who felt that SRS compromised the lectures (QS2): in both cases, the majority of students did not regard SRS as compromising.
Having SRS as a natural and integrated part of the lecture is important as students are wary of how teachers distribute time during lectures, especially regarding SRS. This includes all aspects of SRS, such as how long it takes to set up the system and present the quiz or even how long the students have to vote. During the voting session, there is a 30-second timer during which students can deliver a vote. Several students complained about 30 seconds being too long because “it doesn't take more than a few seconds to vote,” especially when they have already made up their mind, showing that even a few seconds of “dead time” can have a negative impact. Distribution of time is especially important during discussions. The students emphasise that relevant peer discussion is usually maintained only until they come to an agreement about the correct answer. If the discussion time drags on after a consensus is reached, it increases impatience and irritation among the students.
Student 1: It gets boring when it takes too long.
Student 2: Yes, it should have been more effective [in a particular course].
Student 1: It is much worse to sit doing nothing rather than not having enough time to finish.
What constitutes the proper use of time during SRS quizzes greatly depends on the nature of the questions. Factual questions can be beneficial for quick repetition, but they are otherwise regarded as an inefficient use of time with little or no learning gains. According to the students, with such questions they either know the answer or they do not. Consequently, including an individual thinking period before group discussion, which students normally regard as an important factor for good discussions, has little value since there are no reasoning skills involved other than knowing where to look for the answer in the textbook. The discussions themselves also have little learning gains for the same reason. The students therefore emphasise conceptual questions, where the answer is not obvious and a deeper level of reflection is required (making the initial thinking period more valuable), as more beneficial to their learning and therefore a better use of SRS time.
The teachers who are regarded as the most proficient in the use of SRS are also those who use it most frequently, are well prepared, and have a clear purpose for using it. SRS is then regarded as a tool in their lectures. It is also these teachers whom students perceive as most committed and enthusiastic. Commitment and attitudes are extremely contagious; if the teacher shows low commitment and enthusiasm towards using SRS, for example, by regarding it as a chore or by being insufficiently prepared, SRS is experienced as more or less meaningless. Commitment is explained by the students as a “two-way street,” in that SRS is dependent on two parts: the teacher and the students. These two are dependent on each other in order for a lecture with SRS to function properly. If the students sense that the teacher does not care or does not put an effort into making SRS work, their motivation towards participation during SRS quizzes decreases. Students find it motivating to see teachers using SRS actively as a tool to improve the quality of their lectures.
Student 1: The way [teacher] uses SRS, it makes me feel that it is something that [teacher] really wants to use to become better. Yes, both as a teacher and for us to understand it better. [Teacher] uses it really in a way that is supposed to benefit us.
Student 2: Yes, and [teacher] seems very committed and seems very certain of.. yes like he said, prepares herself more. And then you as a student come on the same level as a teacher, really, in the way that when you see that someone who is supposed to teach you something, prepares for the lectures, then it has to be much more fun to be a teacher. And for you as a student, it goes both ways as well. If you as a student see that the teacher comes unprepared, then you lose a lot of motivation around it, especially such things as SRS. If you see that the teacher doesn't care about it or hasn't prepared enough for it, for instance, if there are a lot of mistakes in the questions and such.
Student 3: Plus that it is very motivating to see that [teacher] uses SRS as a tool and not something that she has to use. That is really good.
One student summarised his view of the importance of teacher commitment and how it affects students by drawing a parallel to a working environment, and the authority of the supervisors:
I think that most of us regard this [the preparatory engineering course] as our workplace. We have struggled and worked for this and then we also take it very seriously. And of course at such a workplace, the attitudes of your colleagues and bosses spread to the other employees. And it is like this here as well. So if you picture the teacher as your boss. And then they are, at least in my eyes, looked upon as an authority who we are supposed to look up to and learn from. So if they then, maybe without really meaning it, give us a feeling that they don't really care about learning, or that they don't care about the result of their use of the SRS, then this will spread to us.
There are several factors that give students an impression of low teacher commitment, and insufficient preparation is one of them. Perhaps, the most important aspect of SRS preparation is the actual quizzes: an SRS question can pique students’ interest in the subject matter and motivate them to engage in peer discussion, but it can also be a source of irritation and frustration. According to the students, the requirements for having a good SRS question are higher than those for other questions or problems. As one student put it:
The questions have to be done properly; it is much easier to regard an SRS-question as ridiculous in comparison to other problems.
Questions have to be implemented as a natural part of the lecture. Even if teachers have not made the questions themselves, but have used those in a question bank or from the literature, they still have to make them their own. Students familiarise themselves with the teacher's and textbook's ways of explaining, and therefore it can cause confusion if they are suddenly presented with a question with a very unfamiliar presentation (e.g. different wordings or use of illustrations). The confusion can be further increased if the teachers fail to properly explain the context of the quiz because of lack of preparation. However, this should not be confused with presenting students with questions with an unfamiliar context or setting, which can be an important part of conceptual quizzes.
Also, students emphasise that there has to be a rationale or motivation behind the questions. In other words, the quiz has to have a clear purpose. Is the question testing newly acquired theory, or is it checking if the students remember yesterday's lecture? Moreover, the questions themselves have to be clear. What is actually being asked? If the questions are not clear, students use a lot of time and energy trying to interpret the questions rather than working towards finding the solution. This can be a major source of frustration.
Student 1: Yes, like yesterday.. lately there have been much of.. so much interpretation of the questions that I don't even want to bother. I want pure.. I mean, I want questions that I can read and understand immediately.
Student 2: Yes, that it is clear.. a clear question that has an answer, not where you have to sit and try to figure out what the questions are asking you. You are supposed to mull over the solution, not the question in itself!
One of the benefits of SRS is the opportunity for the teacher to see the level of understanding among the students and use this information to tailor the lectures. However, there are several factors that affect students’ decision making during a discussion. One such factor is opinions from “strong” students (i.e. A students), which are valued higher than the opinions of students with lesser skills. If students feel uncertain, it is very easy to agree on a solution presented by stronger students. This can result in the rest of the group accepting explanations whether they have understood them or not, and the voting results are thus not a correct representation of the level of understanding among the students (see Nielsen, Hansen-Nygård, and Stav 2012, for a more detailed discussion of these results).
Consequently, the students emphasise the importance of the teacher explaining all alternatives, both why the correct alternative is correct and why the incorrect ones are wrong, even though the majority of students have answered the correct alternative. The teacher's explanation is in fact regarded by the students as the most important aspect of SRS with regard to their learning. This was shown in the 2011–2012 survey, where 62% of students answered “teacher's explanation” on the question “What do you experience as the most important aspect of SRS with regard to your own learning?” (the other alternatives were “group discussion before voting” (7%), “experiencing increased engagement and motivation” (9%), the immediate feedback from the voting results (14%), and “other” (8%); N = 160).
Students emphasise that they do not want to guess blindly or vote on an alternative they do not understand as they fear this can mislead the teacher and result in an insufficient explanation of the solution. Therefore, there is one aspect of the questions that has been repeatedly brought up: the inclusion of an alternative called “don't know”. The students regard it as a consolation that the teacher can see that there is a large number of students who are uncertain. Having “don't know” as an alternative in the questions is regarded as a tool with which they can “push” the teacher into giving more in-depth explanations following the discussions. For some students, having this tool is regarded as such an important factor that they will refrain from answering if “don't know” does not feature among the alternatives.
Student 1: I think that people are more inclined to avoid voting if ‘don't know’ is not present.
Student 2: Yes, we often see that there is a difficult question when there are a lower number of people voting.
Student 1: It is natural that it becomes like this when we don't have the opportunity to answer ‘don't know’. We don't want to answer just for the sake of answering, what is the point of that?
We have presented the most significant teacher-centric aspects that we have found negatively affecting students’ experience of SRS after 3 years of testing at Sør-Trøndelag University College in Norway. Students in the preparatory engineering course have a tight time schedule, having to take approximately 2 years of senior high-school curriculum in 1 year, which results in them being extremely focused on what kinds of activities lead to effective learning. While students can easily skip exercises from the textbook, they do not have the same opportunity with SRS quizzes. When a quiz is presented, there is a certain amount of lecture time used on this quiz irrespective of the quality of the questions. The requirements for the SRS questions to be positively received, compared to traditional textbook problems, probably increase for this reason.
The tight time schedule also results in any activity that is ineffective or that leads to spending unnecessary time on SRS (e.g. by fumbling with the software) being prone to cause irritation. Effective use of SRS does not, however, translate to “using less time on each quiz.” During the first year of testing, we compared the peer instruction method with and without the initial thinking period and voting session. Even though including this initial session results in more time spent on each quiz, 70% of the students (N = 109) still preferred having this initial thinking period (with 19% preferring not having it) since this would increase the quality of the discussions.
Students emphasise teacher commitment as the most important factor for successful SRS implementation. Teacher commitment does not directly translate to teachers being enthusiastic when using SRS, though this is a major benefit. Poor preparation, inconsistent use of SRS, not having a clear goal of SRS use, and bad question design all give students an impression of low teacher commitment. This can result in a “vicious circle” where motivation and participation decrease among the students, which, in turn, further decrease the teacher's motivation for using SRS during class – the two-way street discussed in this article.
Teachers with more SRS experience, either by having previous experience with SRS or by simply using it more, were rated higher by the students in our study. However, although the 2011–2012 survey showed those students found SRS to be a less natural and integrated part of the lectures with inexperienced teachers, they did not regard SRS as compromising to the lectures. This may seem contradictory to the attitudes of “use it right or not at all” which were prominent during the interviews of the previous year. A possible explanation for this result could be that SRS was used to a lesser extent in these courses (QS4 in Table 2), and as a result it was not used enough to compromise the lectures. Also, most students who answered the survey had participated in courses with and without teachers with previous SRS experience. Since the students have seen the benefits of using SRS “the right way,” it is possible that this has influenced the survey results from the courses with inexperienced teachers.
The difference between teachers with and without previous SRS experience may have been amplified by the omission of the forum used in the previous year. During the 2010–2011 testing, teachers had regular meetings with one or more of the researchers where all aspects of SRS could be discussed and teachers shared experiences of using it in their lectures. The new SRS teachers of 2011–2012 did not have this opportunity due to time constraints, and consequently they did not have the same level of guidance as in the previous year. This could have hampered the improvement of their SRS skills on both pedagogical and technical levels. In addition, lack of technical support could be a factor that contributes towards decreased teacher commitment (Caldwell 2007). We received feedback from experienced teachers that they missed having the forums of the previous year and that it was difficult to follow-up inexperienced teachers effectively without such an arena for discussion.
During the testing periods, some teachers reported a noticeable amount of students not voting during quizzes. One possibility could be that this was a result of technical difficulties, but the students expressed the omission of “don't know” as an alternative as a major reason for less participation for fear of misleading the teacher by guessing. There was some discussion among the researchers (and, to some extent, the teachers) on whether such an alternative should be included. Although students have high regard for such an alternative, some of the researchers feared that including “don't know” would give students an “easy way out.” Being uncertain about the answer is an important aspect of conceptual SRS questions. The answer should not be obvious, and students have to challenge themselves to use their knowledge about the curriculum in an unfamiliar setting, a task that can be daunting for those students who often rely on memorisation of familiar situations. Including the “don't know” alternative could result in students not making enough effort to challenge themselves to find the solution since they have the option to “give up.”
Another possible scenario, however, could be that, when presented with a difficult quiz, the omission of “don't know” decreases students’ motivation to challenge themselves. The more difficult the question, the higher the probability of students not finding the correct solution or at least not understanding it. Since the students do not wish to guess, they might refrain from trying to find the solution because it is highly improbable that they will vote at that session and therefore they see the effort of working out a solution as a waste of time. The act of giving a vote can be a major motivation for participation during SRS as it gives meaning to both the participation and group discussion (Hansen 2008), but much of this motivation can be lost if students know they are likely to refrain from voting.
Having “don't know” as an alternative is briefly discussed in the study by Draper and Brown (2004), where they reported that few students chose “don't know.” The researchers argued that the anonymity of answering with SRS seemed to make students more inclined to choose a definite answer despite being very uncertain, which is contradictory to the reports of the students in our study. The effects of including “don't know” as an alternative are not clear, and future research could entail comparisons of learning gains (or students’ experiences) from two groups where the only difference in the questions is the inclusion or omission of “don't know.”
The problem of participation is most likely not an issue if the students are rewarded for their participation in the form of points on their grade. Participation in SRS quizzes at Sør-Trøndelag University College during the testing periods did not have any effects on students’ grades, and thus there was no “punishment” for not participating. However, as we mentioned in this article, having SRS participation graded can result in less student satisfaction (Knight and Wood 2005; Reay, Li, and Bao 2008). There have even been results indicating that having extra points for correct quiz answers can lower the quality of the group discussions (James 2006), as opposed to giving points only for participation. Some commercial SRS brands give students the opportunity to follow up their vote with a rate of confidence level (Bruff 2012). Such an option could negate the necessity of a “don't know” alternative as the students would have the opportunity to state their uncertainty.
We end this article with a summary of the recommendations that can be drawn from our results for teachers wanting to use SRS in their lectures. First of all, teachers should have a clear goal and motivation for using SRS in their lectures, and be consistent in their use of SRS towards this goal. A lot of attention should also be given to preparation regarding the methodology, software and quizzes to be used with SRS. Choice of methodology and what quizzes to present will vary based on what the teacher wants to achieve with SRS and a particular quiz. Lack of preparation can cause unnecessary dead time, a major cause of irritation among the students, as well as give students the impression of low teacher commitment (as will lack of consistency or a clear motivation for using SRS). Such an impression can heavily influence students’ attitudes towards SRS. In contrast, students are likely to reward teachers who show commitment and use SRS for the benefit of the students, by being more active when using SRS. Teachers should also be careful when interpreting the voting results, as there are several factors that can influence students’ decision making. Consequently, we recommend always giving a thorough explanation of all alternatives after the voting has ended. If SRS is used for assessing learning, it can also be beneficial to give students the opportunity to state their uncertainty.
Barber, M. & Njus, D. (2007) ‘Clicker evolution: seeking intelligent design’, CBE – Life Sciences Education, vol. 6, pp. 1–8. PubMed Abstract | PubMed Central Full Text | Publisher Full Text
Barnett, J. (2006) ‘Implementation of personal response units in very large lecture classes: student perceptions’, Australasian Journal of Education Technology, vol. 22, pp. 474–494.
Boyle, J. T. & Nicol, D. J. (2003) ‘Using classroom communication systems to support interaction and discussion in large class settings’, Association for Learning Technology Journal, vol. 3, no. 11, pp. 43–57. Publisher Full Text
Bruff, D. (2012) ‘Classroom response systems (“clickers”)’, Vanderbilt University, [online] Available at: http://cft.vanderbilt.edu/teaching-guides/technology/clickers/
Caldwell, J. E. (2007) ‘Clickers in the large classroom: current research and best-practice tips’, CBE – Life Sciences Education, vol. 6, pp. 9–20. PubMed Abstract | PubMed Central Full Text | Publisher Full Text
Charmaz, K. (2001) ‘Grounded theory’, in Rethinking Methods in Psychology, eds. J. A. Smith, R. Harre & L. Langenhove, Sage Publications, London, pp. 27–49.
Charmaz, K. (2003) ‘Grounded theory’, in Qualitative Psychology; A Practical Guide to Research Methods, ed. J. A. Smith, Sage Publications, London, pp. 81–110.
Crouch, A. H. & Mazur, E. (2001) ‘Peer instruction: ten years of experience and results’, American Journal of Physics, vol. 69, no. 9, pp. 970–977. Publisher Full Text
Draper, S. W. (2002) ‘Evaluating effective use of PRS: results of the evaluation of the use of PRS in Glasgow University, October 2001–June 2002’, Glasgow University, [online] Available at: http://www.psy.gla.ac.uk/∼steve/ilig/papers/eval.pdf
Draper, S. W. & Brown, M. I. (2004) ‘Increasing interactivity in lectures using an electronic voting system’, Journal of Computer Assisted Learning, vol. 20, pp. 81–94. Publisher Full Text
Dufrense, R. J. et al. (1996) ‘Classtalk: a classroom communication system for active learning’, Journal of Computing in Higher Education, vol. 7, pp. 3–47. Publisher Full Text
Graham C. R., Tripp, T. R., Seawright, L. & Joeckel, G. (2007) ‘Empowering or compelling reluctant participators using audience response systems’, Active Learning in Higher Education, vol. 8, pp. 233. Publisher Full Text
Hake, R. R. (1998) ‘Interactive-engagement versus traditional methods: a six-thousand student survey of mechanics test data for introductory physics courses’, American Journal of Physics, vol. 66, no. 1, pp. 64–74. Publisher Full Text
Hansen, G. (2008) Men pausen tilførte liksom faget. Studenters opplevelse av bruken av studentresponssystem på forelesningen, Unpublished Master's Thesis, Norwegian University of Science and Technology.
James, M. C. (2006) ‘The effect of grading incentive on student discourse in peer instruction’, American Journal of Physics, vol. 8, pp. 689–691. Publisher Full Text
Kay, H. R. & LeSage, A. (2009) ‘Examining the benefits and challenges of using audience response systems: a review of the literature’, Computers & Education, vol. 53, pp. 819–827.
Knight, J. K. & Wood, W. B. (2005) ‘Teaching more by lecturing less’, Cell Biology Education, vol. 4, pp. 298–310. PubMed Abstract | PubMed Central Full Text | Publisher Full Text
Masikunas, G., Panayiotidis, A. & Burke, L. (2007) ‘The use of electronic voting systems in lectures within business and marketing: a case study of their impact on student learning’, Research in Learning Technology, vol. 15, no. 1, pp. 3–20. Publisher Full Text
Mayer, R. (2005) The Cambridge Handbook of Multimedia Learning, Cambridge University Press, Cambridge.
Mazur, E. (1997) Peer Instruction: A User's Manual, Prentice Hall, Upper Saddle River, NJ.
Nielsen, K. L., Hansen-Nygård, G. & Stav, J. B. (2012) ‘Investigating Peer Instruction: How the initial voting session affects students’ experiences of group discussion’, ISRN Education, vol. 2012, doi: 10.5402/2012/290157. Publisher Full Text
Rao, S. P. & DiCarlo, S. E. (2000) ‘Peer instruction improves performance on quizzes’, Advances in Physiology Education, vol. 24, pp. 51–55. PubMed Abstract
Reay, N. W., Li, P. & Bao, L. (2008) ‘Testing a new voting machine question methodology’, American Journal of Physics, vol. 76, no. 2, pp. 171–178. Publisher Full Text
Rice, R. E. & Bunz, U. (2006) ‘Evaluating a wireless course feedback system: the role of demographics, expertise, fluency, competence, and usage’, Studies in Media & Information Literacy Education, vol. 6, no. 3, pp. 1–32.
Smith, M. K., Wood, U., Krauter, K. & Knight, J. K. (2011) ‘Combining peer discussion with instruction explanation increases student learning from in-class concept questions’, CBE – Life Sciences Education, vol. 10, pp. 55–63. PubMed Abstract | PubMed Central Full Text | Publisher Full Text
University of Wisconsin. (2012) ‘Student response system’, [online] Available at: http://www4.uwm.edu/ltc/srs/faculty/best_practices.cfm