Figure 1. An exam aquarium at TUAS.
Matti Kuikkaa*, Markus Kitolab and Mikko-Jussi Laaksob
aFaculty of Business, ICT and Life Sciences, Turku University of Applied Sciences, Turku, Finland; bDepartment of Information Technology, University of Turku, Turku, Finland
(Received 11 September 2013; final version received 8 September 2014; Published 8 October 2014)
Abstract
Time pressures often necessitate the use of more efficient exam tools, such as electronic exams (e-exams), instead of traditional paper exams. However, teachers may face challenges when introducing e-exams in a higher education context. This paper describes what kinds of challenges teachers may face when introducing e-exams, based on experiences in Turku University of Applied Sciences (TUAS) where e-exams have been used since 2012. For this research, the authors used their personal experiences as administrators and teachers of current e-systems used for e-exams in universities in Turku, Finland. Quantitative data were collected by survey from teachers in TUAS (the case study). The learning management systems, Moodle, Optima and ViLLE, and dedicated e-examination systems, Soft Tutor and Tenttis, were also compared to clarify what kind of features are available in order to ease teachers’ work with examinations. The results identified various challenges during e-exam introduction in TUAS. The paper also provides a list of essential services or features for teachers to ease the introduction of e-exams. Among the analysed systems, ViLLE supported most of the required features, and can be used for both learning management and e-examination systems, providing teachers with one single system, which was found to be important to teachers. The key service found by this paper is ‘support for teachers’, which is in line with previous studies.
Keywords: e-assessment; e-examination; e-exam; teacher; feature
*Corresponding author. Email: matti.kuikka@turkuamk.fi
Research in Learning Technology 2014. © 2014 M. Kuikka et al. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 Unported (CC BY 4.0) License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.
Citation: Research in Learning Technology 2014, 22: 22817 - http://dx.doi.org/10.3402/rlt.v22.22817
Assessment accounts for a significant part of teachers’ workload, especially in large student groups. The use of e-examination could help in addressing this problem, but the introduction of new technology can be challenging.
To investigate these challenges, Turku University of Applied Sciences (TUAS) is used as a case study, which uses Optima as the main learning management system (LMS) and Soft Tutor for (camera-controlled) electronic exams (e-exams). Piloting of e-exams in the camera-controlled environment started in 2012, at which point the main purpose of using e-exams was re-takes. The adoption of e-exams has been quite slow in TUAS, primarily because the teachers decide whether to use e-exams or not, and also because these teachers have faced difficulties in using e-exams. This article thus clarifies the reasons behind this slow uptake and proposes means to overcome the challenges.
The systems used in the University of Turku (UTU) are used here for comparison. UTU uses Moodle and ViLLE for LMSs and Tenttis for (camera-controlled) e-exams. All the LMSs compared are used for e-assessment during courses, and for e-exams in a classroom environment under teacher supervision. The authors, moreover, administer the Soft Tutor and ViLLE systems, and also use Optima, Soft Tutor, Moodle and ViLLE as teachers. Therefore, the research knowledge is also based on practical experience.
The article is structured as follows: first, the key terms used are explained. Then, the challenges faced in TUAS are summarised, followed by the research objectives. A literature review analyses related papers, after which the research section begins by presenting systems used for e-exams in the universities in Turku. The results of the quantitative research concerning the teachers of TUAS are then presented. Finally, the findings are discussed and conclusions are drawn based on the survey results and the comparison of the systems.
Electronic assessment (e-assessment) can be defined as a method where information technology is used for any assessment-related activity (JISC 2007a, p. 6; Rout and Patnaik, 2011, p. 12).
Electronic examination (e-examination) is intended to serve as summative (final) assessment – e-exam – in order to define the evaluation – grade – for a course. Electronic examination can be defined as ‘a system that involves the conduct of examinations through the web or the intranet’ (Ayo et al. 2007, p. 126). Functionally, an e-exam can be provided using a dedicated system or it can be included as a module part of a LMS.
E-assessment types are typically categorised into three groups (Cook and Jenkins 2010, p. 8; Crisp 2011, p. 6; JISC 2007a, p. 6):
Crisp (2011) has introduced a fourth assessment type –integrative assessment –where the purpose is to influence a student’s future learning based on feedback received from a teacher, other students or through self-evaluation.
Exam aquariums are places in which a student can complete an e-exam without staff supervision. The regulation is instead provided by means of recording media, such as video. Students select a time to take an exam in the aquarium. That is, the student makes a reservation and then takes the exam with video control without personnel involvement, see Figure 1. Exam aquariums are typically used for re-take exams and are widely used in universities in Finland. Both TUAS and UTU use exam aquariums.
Figure 1.
An exam aquarium at TUAS.
TUAS is a higher education institute that offers bachelor and master degrees in environmental technologies, healthcare, life sciences, ICT and business with 450 teaching staff (www.tuas.fi).
The main LMS in use in TUAS is Optima. Soft Tutor was introduced in 2012 as the e-exam solution for video-controlled re-take exams instead of re-take exams with pen and paper with personnel control. The project of introducing e-exams started in 2011, run by the main author of this paper. At that time, Tenttis was the most widely used system for e-exams in Finland, followed by Soft Tutor; universities using Moodle as their LMS quite often use Tenttis for their e-exam system. As TUAS did not use Moodle as its LMS, Tenttis was not favoured, and Soft Tutor was selected to be the e-exam system in the exam aquariums instead of Tenttis. Other reasons affecting this selection were the features, lower resource cost and support personnel needs, and because both students and teachers were able to use any kind of attachment for e-exams with Soft Tutor, which was not possible in Tenttis.
Over 1,700 e-exams were run using the Soft Tutor system in 2013. By the end of June 2014, 1,526 e-exams were taken using Soft Tutor. As such, the situation is improving. For comparison, UTU, which uses Tenttis in its exam aquarium, had about 2,500 e-exams taken during 2014. However, the number of teachers actively using the e-exam system in TUAS is still low. About 90 teachers have so far used Soft Tutor e-exams, although most only used the system once or twice. Currently, about 25 teachers actively use Soft Tutor for e-exams.
This article provides answers to the following questions:
E-assessment and e-examination have been studied widely, but mainly from the students’ point of view. However, the following challenges for teachers and subjects affecting the adoption of e-assessment have been identified by previous papers.
The JISC’s publication ‘REAP: Re-engineering Assessment Practices in Scottish Higher Education’ (2007b) states that assessment comprises a large proportion of teachers’ workload and is a major bottleneck in large student groups. The use of new technology could help relieve this problem. By using ICT, assessment can be more student-centric as students by themselves can identify gaps in their learning. More benefits are achieved with ICT when courses are redesigned with improved pedagogical objectives in mind (JISC 2007b, pp. 15–16).
E-assessment not only reduces the time needed to mark exams but also widens the variety of methods available for assessment compared to traditional pen and paper exams (JISC 2007a, p. 16). Students also feel that e-assessment improves their learning. In addition, technology enables teachers to better monitor group interactions on-line as well as in the classroom (JISC 2007b, p. 10).
Crisp’s ‘Teacher’s Handbook on e-Assessment’ (2011) states that digital environments need to be used in the future instead of traditional sessions in classes even if they do not provide efficiency improvements compared to traditional methods with pen and paper. Wider usage of digital environments push this change due to new skills and capabilities needed in the twenty-first century. However, there is a challenge in that majority of teachers do not have the competences required to use new technological innovations (Crisp 2011, p. 12).
When deciding on how to proceed with e-assessment, the time needed to create e-assessment questions was found to be a barrier, based on the research by Whitelock and Brasher (2006, p. 15). Therefore, it is essential to provide time and support for those doing such work (Crisp 2011, p. 12; JISC 2007a, p. 25, 2007b, pp. 16–17). Moreover, automatic assessment is an important benefit for teachers, as it saves time. Due to this timesaving, teachers could devote more effort into improving the contents of the exam questions (Bernik and Jereb 2006, p. 145; Crisp 2011, p. 12; JISC 2007a, pp. 24–25).
Crisp’s handbook (2011) states that plenty of guidelines are available on how to use new technology in education, but a lack of teachers’ time and financial resources decrease the possibilities to widen the usage of e-assessment. However, e-assessment should not increase the workload of teachers in the long run.
Crisp (2011) recognises that additional effort is needed during the introduction of e-assessment. When introducing e-assessment, a project needs to be in place. This project should clarify the characteristics of possible assessment software tools. Resource and competence requirements also need to be clarified, and the e-assessment process and organisation to support e-assessment need to be in place (Cook and Jenkins 2010; JISC 2007b, p. 16).
In order to ensure better results and gains, redesigns of assessment should be done by teams instead of individual teachers. The involvement of all members of a course design team is recommended. Success stories can then be spread across the organisation and staff should be available to support and participate in the redesign. Such support is important to academic staff, as most do not have the time or skills to carry out the work during the piloting phase (JISC 2007b, pp. 5, 16–17).
It is also important to have organisational level commitment for e-assessment. If original drivers move on, the support for changes may collapse. Senior management support is thus vital. ICT should enable enhanced learning and teaching, and not just increase staff workload and costs. Therefore, it is essential to define the pedagogical purpose of each ICT application (JISC 2007b, pp. 7, 15–17).
Staff training is critical in order to guarantee quality of assessment on a wide scale. Dedicated support staff is needed in order to ensure support for and reliability of the solution. Dedicated staff may also take full responsibility for assessment development for teachers. Technical support should provide guidance, also covering instructions for using the tools. Failure to find this support may cause serious motivation problems among teachers. It is thus required that such support should be available in the organisation in the long term (JISC 2007a, p. 18, 2007b, p. 16).
Due to technological developments, computer-based problems have decreased over the past years. However, the interoperability of e-assessment systems and other systems can cause problems. Standardised interfaces are needed to guarantee interoperability (JISC 2007a, p. 14). Such interoperability problems have been recently recognised by Brink and Lautenbach (2011, pp. 507–508), who found that lecturers had problems not only importing data from one campus to another but also problems when viewing results.
Indeed, knowledge of IT and computer skills are among the prerequisites for accepting e-examination (Adegbija, Fakomogbon, and Daramola 2012, p. 65). Similarly, Osang (2012, p. 307) found a positive relation between teachers’ computer skills and the feedback they provided about e-examination. Moreover, teaching styles affect how lecturers use the e-assessment tools. Therefore, it is important to guide lecturers on how to use the tools (Adegbija, Fakomogbon, and Daramola 2012, p. 65; Brink and Lautenbach 2011, pp. 507–508). There needs to be a defined pedagogical framework for assessment redesign using ICT (JISC 2007b, p. 16).
Usability affects the acceptance of e-examinations, as seen in several studies. Gilbert et al. (2009, p. 12) stated that usability is an important factor for teachers when starting to use (new) electronic systems. A new user interface may make it difficult for teachers to adopt the system. Rout and Patnaik (2011, p. 19) stated that from a usability perspective, the creation of the first positive experience of e-examination had a positive influence on the adoption of e-exams. In line with the above, Crisp (2011, p. 16) stated that teachers need simple tools when developing e-assessment methods.
Cook and Jenkins’s report ‘Getting Started with e-Assessment’ states that there are challenges in designing questions for e-exams. It is difficult to create non-objective questions and questions to measure students’ in-depth understanding. There may also be additional costs for setting up these tools, defining questions and exams, and making sure the system works (Cook and Jenkins 2010, pp. 3–4). The actual layout of questions also affects their quality. Lecturers should thus have the means to test how an exam is seen and completed from a student’s point of view (Brink and Lautenbach 2011, pp. 507–508).
If exam questions are used in several universities, there may be a need to define ‘standardised’ questions. These classified standardised questions could enable better objectivity for the evaluation and make the questions comparable in terms of difficulty. Recently, studies have been conducted into the complexity of exam questions, and methods for creating standardised exam questions for introductory courses in programming (Sheard et al. 2013, p. 145–154).
Copyright issues for questions banks may also cause concerns among teachers as the questions may even be shared between institutions (JISC 2007a, p. 9). For instance, in the ViLLE system used at UTU, all questions are visible to all users. However, usage of e-assessment and sharing of efforts is vital in order to manage larger groups with smaller resources. Recognising the added value of e-assessment is necessary in order for the change from using traditional methods to be successful (JISC 2007a, pp. 24–26).
The research began with a presentation of systems currently used for e-exams in the universities in Turku. The LMSs Moodle, Optima and ViLLE, and e-examination systems Soft Tutor and Tenttis are compared. These systems are the major systems used as learning environments or for e-examination in higher education in Finland. This part of the research is mainly descriptive. Here, we use available documentation, feedback received from teachers, and personal experiences when administrating systems to make the comparison.
The second part of the research was conducted through a questionnaire. Based on the comparison and the authors’ experiences, a survey was created. The survey was sent to teachers with accounts to use Soft Tutor at TUAS during autumn 2013 to clarify their opinions and challenges they have with e-exams. The survey was also available via the intranet of the university. In total, 48 teachers responded: 22 had been using e-exams. The results of this survey were used in order to clarify teachers’ views about the examination. These results were then compared with previous research and the authors’ experiences.
The following systems supporting examination features are analysed in this report:
Moodle and Optima are LMS, where the main purpose is to provide an environment where teachers can maintain course materials and assignments. These systems support features for examination purposes, but the systems are not actually intended for e-exams. Moodle is an open source program, and Optima is a commercial solution.
Soft Tutor and Tenttis are dedicated systems for e-examination. Tenttis is developed based on Moodle with a simplified teacher interface for exercises. Soft Tutor is a commercial solution. E-exams with these systems are mainly conducted in dedicated locations (an exam aquarium) with video control, and are mainly used for re-take exams.
ViLLE was created to be both an LMS and e-examination system from the outset, originally for programming. E-examination is thus one function among others in the system. E-exams through ViLLE are done in a classroom environment. TUAS uses Optima as an LMS and Soft Tutor for video-controlled e-exams. Some teachers use also Optima and ViLLE for e-exams in classrooms. UTU uses Moodle and ViLLE as LMS and Tenttis for video-controlled e-exams. ViLLE is used for e-exams in classrooms.
Moodle is an open source e-learning platform. Moodle was originally designed by Martin Dougiamas and the first version was released in 2002. Today, Moodle has about 70,000 registered sites with 65,000,000 users. Moodle has no separate exam feature, but does contain tasks and objects that could be used for e-assessment.
Optima is a learning environment provided by Discendum (www.discendum.com), and was introduced in the beginning of 2000. There is an exam feature included in the system, in which some basic question types with automatic assessment are available. Optima also contains features like return boxes, surveys, discussion lists and diaries that could be used for assessment while learning.
Soft Tutor is an on-line application that is accessible via a web browser. Teachers must have their own credentials in order to use the system. Students receive a one-time username and password when signing up to an exam. The locations from which the exam can be taken are controlled by IP addresses, and several locations can be defined for the system per exam by the teacher. The teachers’ interface contains means to maintain exams and follow students’ signups and exams. This solution was originally designed in 2006, and no significant changes have been made recently. The system is provided by Sordino systems (www.sordino.fi), which also takes care of technical support and provides the credentials.
Tenttis in an on-line application that is accessible via a web browser. This solution is based on Moodle and was originally created around 2006 and maintained by the University of Tampere. CSC – the IT Center for Science Ltd. of the Ministry of Education, Science and Culture – has taken over the development of the system. The current version of Tenttis is in maintenance mode. The system can either be run in a local server in the school or remotely using the system maintained by CSC.
ViLLE is a collaborative education platform which makes it possible for teachers to easily create virtual courses and automatically assess exercises of different kinds. All created material can be utilised, commented on and evaluated by other teachers. The server for this solution is located in the Department of Information Technology in the University of Turku. The solution was introduced 2004. Teachers and students can use their normal school credentials or local user IDs to access the system. ViLLE is an on-line application which is accessible via a web browser, but can also be used as a local server without an external network. ViLLE has been shown to provide improvement in learning in the area of programming by means of visualisation (Laakso 2010; Laakso et al. 2008, pp. 134–135).
The survey was administrated to teachers in TUAS during the autumn of 2013. The survey was sent to around 60 teachers with accounts in Soft Tutor at TUAS during the autumn of 2013 to clarify the challenges and needs they have with e-exams. The survey was also available via the TUAS intranet. By the end of November 2013, 48 responses were received, and 22 respondents had used e-exams. The following questions included in the survey are analysed in this paper:
Teachers were asked to evaluate the importance of features and characteristics in the e-exam systems on a scale from 1 (unimportant) to 5 (very important). Table 1 contains the results and shows the analysed systems supporting the features.
The results show that usage of e-exams should decrease teachers’ workload (b). Teachers prefer to use their normal school credentials for e-exams (c). Teachers also like to define the evaluation criteria for open questions (g). The majority of respondents also thought it was important to have the possibility to receive feedback from other teachers using his/her exercises (f). Teachers stated that it was important that students should receive immediate feedback after the e-exam was completed (h).
Multiple-choice questions were the most wanted automatically evaluated question type. Other typical automatically evaluated question types: gap filling, connect pairs, short answer and arrange in order were also favoured (Table 2).
| Supported by | ||||||
| Question type | Frequency (%) | Moodle | Optima | Soft Tutor | Tenttis | ViLLE |
| Essay | 35.42 | – | – | – | – | – |
| Definition | 33.33 | – | – | – | – | – |
| Multiple choice | 87.50 | X | X | X | X | X |
| Fill gap | 58.33 | X | X | X | X | X |
| Short answer | 50.00 | X | – | – | X | X |
| Connect pairs | 54.17 | X | – | X | X | X |
| Arrange in order | 47.92 | X | – | X | X | X |
| Programming | 18.75 | X* | – | – | – | X |
| Mathematics | 35.42 | X | – | – | – | X |
It was interesting to recognise that teachers quite broadly wanted essay, definition and mathematical questions to be automatically evaluated (one third of respondents). Teachers commented through the option ‘other’ that questions including pictures or videos also need to be automatically evaluated.
The results of teachers’ opinions about other features in e-exams are shown in Table 3.
| Supported by | ||||||||||||
| Feature | 1 | 2 | 3 | 4 | 5 | Moodle | Optima | Soft Tutor | Tenttis | ViLLE | ||
|
14.58% | 16.67% | 10.42% | 43.75% | 14.58% | X | X | |||||
|
2.08% | 18.75% | 20.83% | 41.67% | 16.67% | X | ||||||
|
2.08% | 10.42% | 4.17% | 37.50% | 45.83% | X | X | |||||
|
8.33% | 10.42% | 10.42% | 35.42% | 35.42% | X | X | X | ||||
|
4.17% | 4.17% | 14.58% | 31.25% | 45.83% | X | X | X | ||||
|
2.08% | 0.00% | 0.00% | 14.58% | 83.33% | X | X | |||||
|
2.08% | 2.08% | 6.25% | 27.08% | 62.50% | X | ||||||
|
4.17% | 8.33% | 20.83% | 14.58% | 52.08% | X | X | |||||
The results show that teachers believe students should be able to select the time to take the exam (c), but it should also be possible for teachers to register students in e-exams (d). Teachers also felt they should be able to set a maximum number of attempts per exam (e). The most important feature was the possibility to test the exam easily (f). Teachers also wished to see statistics about exams both at indiviudal student and group levels (g).
Teachers were asked to state what additional materials they would like to attach to exam questions. Table 4 shows that most teachers preferred to attach files. However, it is noteworthy that one third of the respondents wished to have open Internet access for students during an e-exam.
| Supported by | ||||||
| Material | Frequency (%) | Moodle | Optima | Soft Tutor | Tenttis | ViLLE |
| Attachment file containing text and/or picture (for example an article or drawing) | 81.25 | X | X | X | X | |
| Attachment file containing video or audio material | 52.08 | X | X | |||
| Video or audio material as Internet link | 56.25 | X | X | X | ||
| Open Internet access (when student is able to use Internet during exam) | 35.42 | X | X | X | X | |
The survey also included open questions. Based on these questions, suggestions for the following other services, features or comments were provided by (at least three) teachers:
The teachers’ challenges, based on the survey, are in line with what was found in previous studies. Teachers mentioned subject areas such as the availability of support, need for additional time for e-exam creation, usability and reliability of the system to be important. Previous research stated that the introduction of e-assessment needs to be well planned. This is in line with the survey results. In addition, the survey revealed that teachers feel that there are too many systems for assessment in use. That is, they preferred to have a single system (LMS) providing the functionalities of a learning environment and electronic exam.
The resistance seen by teachers using e-exams may be caused by the change as such: teachers do not like to change their examination habits, as stated in the survey result. Better support and training could help them to overcome this challenge. TUAS has arranged for regular (bi-monthly) teacher e-exam workshops to boost usage, and quite a few teachers join in these workshops. There are also documentation and guidance videos available for how to use e-exams. However, in addition to such support and training, there is a need for management commitment to push the usage of e-exams, as stated in previous papers.
The currently used LMS (Moodle, Optima and ViLLE) and dedicated e-exam solutions (Soft Tutor and Tenttis) provide decent support for e-exams. The widest support is available in ViLLE. However, none of the compared systems provide all the needs that teachers have in one single system. LMS should be developed in such a direction that the very same system could be also used for e-exams in both teacher and video-controlled environments.
Based on the survey, the following services and features are important to teachers when starting to use e-exams:
As mentioned in previously written handbooks and reports about e-assessment, it is vital to have the organisation in place to provide guidance to teachers, not only technically but also pedagogically. TUAS currently has three people supporting teachers, but their main focus has been technical. This needs to be changed.
Based on previous work and the survey results, teachers like to have systems that are easy to use; usability is a crucial aspect for them. A complex interface may cause a teacher to stop using systems. Using the same system for e-exams and LMS also simplifies a teacher’s work environment. There should also be easy test functionality to help teachers verify the correctness of questions. Teachers wish to be able to easily test exam and exercises prepared, as seen in the survey results.
The need to acquire separate credentials for e-exams increases the difficulty of their adoption for teachers. Thus, being able to use existing credentials was seen to be highly required in the survey results. Teachers should be able to use the very same username and password for the e-exam system as used for the network (LAN) login.
Previous research suggests that the amount of time to evaluate exercises is reducing. This is in line with the survey results here that e-exams should reduce teachers’ workload. Automatic evaluation is one obvious timesaving feature. This should also be supported for open questions, as the evaluation of open questions is time consuming. Automatic evaluation for open questions might be difficult to achieve. Therefore, it is important that teachers are able to set grading criteria that could be used either during the evaluation process by the teacher or for self-evaluation by students.
Feedback is important for students in order to enable learning and in order to reduce evaluation time by teachers. The correctness of an answer should be presented to the student directly after the task performance without teacher involvement (for automatically evaluated questions). This feature supports the challenges teachers face with larger groups.
The possibility to make questions visible to other teachers is important. Teachers should be able to share questions flexibly, making it possible for anyone to re-use questions in their courses. Questions should be tagged by subject/competences, enabling other teachers to find them. The possibility to receive feedback from other teachers if they use your questions was also seen as a preference in the survey results.
A reservation calendar could enable students to independently reserve a time to take exams without teacher involvement, in addition to exams in class. Teachers should also be able to register students for exams.
Statistical functions ease teachers’ work in following the progress of students. This functionality is not currently available in the Soft Tutor e-exam, but has been widely required by teachers. Teachers should be able to see information such as the number of attempts, but also see how many times a certain student has taken exams or exercises. It should also be possible to view the duration of a student’s exam attempt.
Using additional materials, teachers are able to create more advanced questions. Teachers should thus be able to attach to exams files containing articles or figures. Teachers also required video and audio materials to be attached in exams (possibly via an Internet link).
Although the participants of the survey were selected by convenient sampling, the reliability of the results is in line with previous papers. The results are also in line with the experience of the authors while providing teacher support for e-exams in TUAS. The small size (48 of 450 teachers) of the sample seemed not to affect the result. However, additional research is needed once there has been a wider adoption of e-exams.
Learning systems are not necessarily created based on teachers’ needs but based on the creativity of developers. An approach where systems are developed starting from pedagogical and teaching perspectives – how to teach new skills – should be promoted. This eases teachers’ entry into using the systems, and creates systems with better usability.
Timesaving is crucial for teachers. Features such as automatic evaluation and sharing of exercises help teachers save time. In order to utilise this fully, teachers can no longer work by keeping their questions private but instead co-operation between teachers is necessary. Moreover, in order to get teachers to use e-exam more widely, it is vital to provide support and to reserve enough time for them for the introduction of e-examination. User-friendly systems may help the adoption phase. It is also recommended that teachers and organisations use the guidance and documentation provided by JISC (http://www.jisc.ac.uk/guides) and Transforming Assessment (http://www.transformingassessment.com) when introducing e-exams or e-assessment.
Of the systems here analysed, ViLLE supports most of the features preferred by teachers. It is the system that can support features for both LMS and e-exams, enabling teachers to have one single system to work with e-exams and learning. This simplifies a teacher’s ICT environment. However, further research should be done in order to discover how a single system can help teachers with their challenges, covering all assessment types. Methods for automatic evaluation of open questions also need further research. Research is also needed in order to clarify the differences of views between current active e-exam users and non-active users.
Adegbija, M. V., Fakomogbon, M. A. & Daramola, F. O. (2012) ‘The new technologies and the conduct of e-examinations: a case study of National Open University of Nigeria’, British Journal of Science, vol. 3, pp. 59–66.
Ayo, C. K., et al., (2007) ‘The prospects of e-examination implementation in Nigeria’, Turkish Online Journal of Distance Education, vol. 8, pp. 125–134.
Bernik, I. & Jereb, E. (2006) ‘Students’ readiness for electronic examinations’, Proceedings of the 5th WSEAS International Conference on Education and Educational Technology, Tenerife, pp. 142–145.
Brink, R. & Lautenbach, G. (2011) ‘Electronic assessment in higher education’, Educational Studies, vol. 37.5, pp. 503–512.
Cook, J. & Jenkins, V. (2010) Getting Started with E-assessment. University of Bath, Bath.
Crisp, G. (2011). Teacher’s Handbook on e-Assessment, Transforming Assessment-An ALTC Fellowship Activity 18, [online] Available at: http://www.transformingassessment.com/moodle/file.php/84/Handbook_for_teachers.pdf
Gilbert, L., et al., (2009). Report on Summative E-Assessment Quality (REAQ), University of Southampton, Southampton.
JISC. (2007a) Effective Practice with e-Assessment: An Overview of Technologies, Policies and Practices in Further and Higher Education, [online] Available at: http://www.jisc.ac.uk/media/documents/themes/elearning/effpraceassess.pdf
JISC. (2007b). Transformation Story: Re-engineering Assessment Practices in Scottish Higher Education (REAP), [online] Available at: http://www.jisc.ac.uk/media/documents/programmes/elearningsfc/sfcbookletreap.pdf
Laakso, M.-J. (2010). Promoting Programming Learning. Engagement, Automatic Assessment with Immediate Feedback in Visualizations. Turku Centre Computer Science. TUCS Dissertations No 131, November 2010, [online] Available at: http://www.doria.fi/bitstream/handle/10024/66222/TUCSDissertations131.pdf
Laakso, M.-J., et al., (2008) ‘The impact of prior experience in using a visualization tool on learning to program’, Appeared in Cognition and Exploratory Learning in Digital Age (CELDA 2008), Freiburg, Germany, Oct 13–15, pp. 129–136.
Osang, F. (2012) ‘Electronic examination in Nigeria, academic staff perspective – case study: National Open University of Nigeria (NOUN)’, International Journal of Information and Education Technology, vol. 2, pp. 304–307.
Rout, G. & Patnaik, S. (2011) ‘A case study on e-examination in Universities of Odisha’, International Journal of Internet Computing (IJIC), vol. 1, no. 2, pp. 12–20.
Sheard, J., et al., (2013) ‘How difficult are exams? A framework for assessing the complexity of introductory programming exams’, Proceedings of the 15th Australasian Computing Education Conference, ACS, Adelaide, SA, pp. 145–154.
Whitelock, D. M. & Brasher, A. (2006) ‘Developing a roadmap for e-assessment: which way now?’ Proceedings of the 10th CAA International Computer Assisted Assessment Conference, The Open University, pp. 487–501, [online] Available at: http://oro.open.ac.uk/11950/1/