Electronic assessment (e-assessment) can be defined as a method where information technology is used for any assessment-related activity (JISC 2007a, p. 6; Rout and Patnaik, 2011, p. 12).

Electronic examination (e-examination) is intended to serve as summative (final) assessment – e-exam – in order to define the evaluation – grade – for a course. Electronic examination can be defined as ‘a system that involves the conduct of examinations through the web or the intranet’ (Ayo et al. 2007, p. 126). Functionally, an e-exam can be provided using a dedicated system or it can be included as a module part of a LMS.

E-assessment types are typically categorised into three groups (Cook and Jenkins 2010, p. 8; Crisp 2011, p. 6; JISC 2007a, p. 6):

Diagnostic assessment: assessment at the beginning of the course to gauge the knowledge levels of students

Crisp (2011) has introduced a fourth assessment type –integrative assessment –where the purpose is to influence a student's future learning based on feedback received from a teacher, other students or through self-evaluation.

Exam aquariums are places in which a student can complete an e-exam without staff supervision. The regulation is instead provided by means of recording media, such as video. Students select a time to take an exam in the aquarium. That is, the student makes a reservation and then takes the exam with video control without personnel involvement, see Figure 1. Exam aquariums are typically used for re-take exams and are widely used in universities in Finland. Both TUAS and UTU use exam aquariums.

TUAS is a higher education institute that offers bachelor and master degrees in environmental technologies, healthcare, life sciences, ICT and business with 450 teaching staff (www.tuas.fi).

The main LMS in use in TUAS is Optima. Soft Tutor was introduced in 2012 as the e-exam solution for video-controlled re-take exams instead of re-take exams with pen and paper with personnel control. The project of introducing e-exams started in 2011, run by the main author of this paper. At that time, Tenttis was the most widely used system for e-exams in Finland, followed by Soft Tutor; universities using Moodle as their LMS quite often use Tenttis for their e-exam system. As TUAS did not use Moodle as its LMS, Tenttis was not favoured, and Soft Tutor was selected to be the e-exam system in the exam aquariums instead of Tenttis. Other reasons affecting this selection were the features, lower resource cost and support personnel needs, and because both students and teachers were able to use any kind of attachment for e-exams with Soft Tutor, which was not possible in Tenttis.

Over 1,700 e-exams were run using the Soft Tutor system in 2013. By the end of June 2014, 1,526 e-exams were taken using Soft Tutor. As such, the situation is improving. For comparison, UTU, which uses Tenttis in its exam aquarium, had about 2,500 e-exams taken during 2014. However, the number of teachers actively using the e-exam system in TUAS is still low. About 90 teachers have so far used Soft Tutor e-exams, although most only used the system once or twice. Currently, about 25 teachers actively use Soft Tutor for e-exams.

This article provides answers to the following questions:

What challenges and wishes do teachers have for e-exams?

The JISC's publication ‘REAP: Re-engineering Assessment Practices in Scottish Higher Education’ (2007b) states that assessment comprises a large proportion of teachers’ workload and is a major bottleneck in large student groups. The use of new technology could help relieve this problem. By using ICT, assessment can be more student-centric as students by themselves can identify gaps in their learning. More benefits are achieved with ICT when courses are redesigned with improved pedagogical objectives in mind (JISC 2007b, pp. 15–16).

E-assessment not only reduces the time needed to mark exams but also widens the variety of methods available for assessment compared to traditional pen and paper exams (JISC 2007a, p. 16). Students also feel that e-assessment improves their learning. In addition, technology enables teachers to better monitor group interactions on-line as well as in the classroom (JISC 2007b, p. 10).

Crisp's ‘Teacher's Handbook on e-Assessment’ (2011) states that digital environments need to be used in the future instead of traditional sessions in classes even if they do not provide efficiency improvements compared to traditional methods with pen and paper. Wider usage of digital environments push this change due to new skills and capabilities needed in the twenty-first century. However, there is a challenge in that majority of teachers do not have the competences required to use new technological innovations (Crisp 2011, p. 12).

When deciding on how to proceed with e-assessment, the time needed to create e-assessment questions was found to be a barrier, based on the research by Whitelock and Brasher (2006, p. 15). Therefore, it is essential to provide time and support for those doing such work (Crisp 2011, p. 12; JISC 2007a, p. 25, 2007b, pp. 16–17). Moreover, automatic assessment is an important benefit for teachers, as it saves time. Due to this timesaving, teachers could devote more effort into improving the contents of the exam questions (Bernik and Jereb 2006, p. 145; Crisp 2011, p. 12; JISC 2007a, pp. 24–25).

Crisp's handbook (2011) states that plenty of guidelines are available on how to use new technology in education, but a lack of teachers’ time and financial resources decrease the possibilities to widen the usage of e-assessment. However, e-assessment should not increase the workload of teachers in the long run.

Crisp (2011) recognises that additional effort is needed during the introduction of e-assessment. When introducing e-assessment, a project needs to be in place. This project should clarify the characteristics of possible assessment software tools. Resource and competence requirements also need to be clarified, and the e-assessment process and organisation to support e-assessment need to be in place (Cook and Jenkins 2010; JISC 2007b, p. 16).

In order to ensure better results and gains, redesigns of assessment should be done by teams instead of individual teachers. The involvement of all members of a course design team is recommended. Success stories can then be spread across the organisation and staff should be available to support and participate in the redesign. Such support is important to academic staff, as most do not have the time or skills to carry out the work during the piloting phase (JISC 2007b, pp. 5, 16–17).

It is also important to have organisational level commitment for e-assessment. If original drivers move on, the support for changes may collapse. Senior management support is thus vital. ICT should enable enhanced learning and teaching, and not just increase staff workload and costs. Therefore, it is essential to define the pedagogical purpose of each ICT application (JISC 2007b, pp. 7, 15–17).

Staff training is critical in order to guarantee quality of assessment on a wide scale. Dedicated support staff is needed in order to ensure support for and reliability of the solution. Dedicated staff may also take full responsibility for assessment development for teachers. Technical support should provide guidance, also covering instructions for using the tools. Failure to find this support may cause serious motivation problems among teachers. It is thus required that such support should be available in the organisation in the long term (JISC 2007a, p. 18, 2007b, p. 16).

Due to technological developments, computer-based problems have decreased over the past years. However, the interoperability of e-assessment systems and other systems can cause problems. Standardised interfaces are needed to guarantee interoperability (JISC 2007a, p. 14). Such interoperability problems have been recently recognised by Brink and Lautenbach (2011, pp. 507–508), who found that lecturers had problems not only importing data from one campus to another but also problems when viewing results.

Indeed, knowledge of IT and computer skills are among the prerequisites for accepting e-examination (Adegbija, Fakomogbon, and Daramola 2012, p. 65). Similarly, Osang (2012, p. 307) found a positive relation between teachers’ computer skills and the feedback they provided about e-examination. Moreover, teaching styles affect how lecturers use the e-assessment tools. Therefore, it is important to guide lecturers on how to use the tools (Adegbija, Fakomogbon, and Daramola 2012, p. 65; Brink and Lautenbach 2011, pp. 507–508). There needs to be a defined pedagogical framework for assessment redesign using ICT (JISC 2007b, p. 16).

Usability affects the acceptance of e-examinations, as seen in several studies. Gilbert et al. (2009, p. 12) stated that usability is an important factor for teachers when starting to use (new) electronic systems. A new user interface may make it difficult for teachers to adopt the system. Rout and Patnaik (2011, p. 19) stated that from a usability perspective, the creation of the first positive experience of e-examination had a positive influence on the adoption of e-exams. In line with the above, Crisp (2011, p. 16) stated that teachers need simple tools when developing e-assessment methods.

Cook and Jenkins's report ‘Getting Started with e-Assessment’ states that there are challenges in designing questions for e-exams. It is difficult to create non-objective questions and questions to measure students’ in-depth understanding. There may also be additional costs for setting up these tools, defining questions and exams, and making sure the system works (Cook and Jenkins 2010, pp. 3–4). The actual layout of questions also affects their quality. Lecturers should thus have the means to test how an exam is seen and completed from a student's point of view (Brink and Lautenbach 2011, pp. 507–508).

If exam questions are used in several universities, there may be a need to define ‘standardised’ questions. These classified standardised questions could enable better objectivity for the evaluation and make the questions comparable in terms of difficulty. Recently, studies have been conducted into the complexity of exam questions, and methods for creating standardised exam questions for introductory courses in programming (Sheard et al. 2013, p. 145–154).

Copyright issues for questions banks may also cause concerns among teachers as the questions may even be shared between institutions (JISC 2007a, p. 9). For instance, in the ViLLE system used at UTU, all questions are visible to all users. However, usage of e-assessment and sharing of efforts is vital in order to manage larger groups with smaller resources. Recognising the added value of e-assessment is necessary in order for the change from using traditional methods to be successful (JISC 2007a, pp. 24–26).

Moodle is an open source e-learning platform. Moodle was originally designed by Martin Dougiamas and the first version was released in 2002. Today, Moodle has about 70,000 registered sites with 65,000,000 users. Moodle has no separate exam feature, but does contain tasks and objects that could be used for e-assessment.

Teacher can use the school's credentials to access the system.

Creation of an exam or drill requires few steps but there are only three types of questions to choose from.

Teachers are not able to use their school's normal credentials when logging in to the system.

Teachers can use the school's credentials to access the system.

Teachers can use their school's credentials or local user IDs to access the system.

Teachers were asked to evaluate the importance of features and characteristics in the e-exam systems on a scale from 1 (unimportant) to 5 (very important). Table 1 contains the results and shows the analysed systems supporting the features.

The results show that usage of e-exams should decrease teachers’ workload (b). Teachers prefer to use their normal school credentials for e-exams (c). Teachers also like to define the evaluation criteria for open questions (g). The majority of respondents also thought it was important to have the possibility to receive feedback from other teachers using his/her exercises (f). Teachers stated that it was important that students should receive immediate feedback after the e-exam was completed (h).

Multiple-choice questions were the most wanted automatically evaluated question type. Other typical automatically evaluated question types: gap filling, connect pairs, short answer and arrange in order were also favoured (Table 2).

It was interesting to recognise that teachers quite broadly wanted essay, definition and mathematical questions to be automatically evaluated (one third of respondents). Teachers commented through the option ‘other’ that questions including pictures or videos also need to be automatically evaluated.

The results of teachers’ opinions about other features in e-exams are shown in Table 3.

The results show that teachers believe students should be able to select the time to take the exam (c), but it should also be possible for teachers to register students in e-exams (d). Teachers also felt they should be able to set a maximum number of attempts per exam (e). The most important feature was the possibility to test the exam easily (f). Teachers also wished to see statistics about exams both at indiviudal student and group levels (g).

Teachers were asked to state what additional materials they would like to attach to exam questions. Table 4 shows that most teachers preferred to attach files. However, it is noteworthy that one third of the respondents wished to have open Internet access for students during an e-exam.

The survey also included open questions. Based on these questions, suggestions for the following other services, features or comments were provided by (at least three) teachers:

(9) Good support, guidance and training for e-exams should be available.

The teachers’ challenges, based on the survey, are in line with what was found in previous studies. Teachers mentioned subject areas such as the availability of support, need for additional time for e-exam creation, usability and reliability of the system to be important. Previous research stated that the introduction of e-assessment needs to be well planned. This is in line with the survey results. In addition, the survey revealed that teachers feel that there are too many systems for assessment in use. That is, they preferred to have a single system (LMS) providing the functionalities of a learning environment and electronic exam.

The resistance seen by teachers using e-exams may be caused by the change as such: teachers do not like to change their examination habits, as stated in the survey result. Better support and training could help them to overcome this challenge. TUAS has arranged for regular (bi-monthly) teacher e-exam workshops to boost usage, and quite a few teachers join in these workshops. There are also documentation and guidance videos available for how to use e-exams. However, in addition to such support and training, there is a need for management commitment to push the usage of e-exams, as stated in previous papers.

The currently used LMS (Moodle, Optima and ViLLE) and dedicated e-exam solutions (Soft Tutor and Tenttis) provide decent support for e-exams. The widest support is available in ViLLE. However, none of the compared systems provide all the needs that teachers have in one single system. LMS should be developed in such a direction that the very same system could be also used for e-exams in both teacher and video-controlled environments.

Based on the survey, the following services and features are important to teachers when starting to use e-exams:

Although the participants of the survey were selected by convenient sampling, the reliability of the results is in line with previous papers. The results are also in line with the experience of the authors while providing teacher support for e-exams in TUAS. The small size (48 of 450 teachers) of the sample seemed not to affect the result. However, additional research is needed once there has been a wider adoption of e-exams.