Using computer-based tests for information science

Computer-based tests have been used extensively in the Department of Information Science at the University of Portsmouth, both for end-of-course examinations and continuous assessment. This paper details the use of computer-based objective testing as an innovative technique for traditional assessment, and the separate problems of continuous computer-aided assessment. Results from three years of research have led to plans for future developments within the department, and the paper provides a checklist of considerations regarded as crucial.

The introduction of objective testing using computer software does not necessarily represent innovative assessment.Where tests occur as an add-on to a course, are timeconstrained, closed-book, invigilated, and where there is little (or no) feedback of results to the students, such testing is best regarded as an innovative technique for traditional summative assessment.A computer-based examination of this nature using the commercial software Question Mark has been operating for a number of years in the Department of Information Science at Portsmouth, in the second-year unit for Logic Programming, with student numbers up to 160.
Computer-based examinations, like written examinations and coursework, can test some things and not others.Their big disadvantage is that answers to the questions have to be simple, either straightforward choices or unambiguous character strings.It is not possible, in the present state of computer technology, to test a student's skill in extended, free-flowing answers.Higher abilities (such as analysis, synthesis, evaluation, and extrapolation of knowledge into new areas) are the hardest to test -some would say impossible to test -on a computer.But the contents of Table 1 show that it is possible to assess everything one might want to assess using a judicial mix of coursework and computer-based examination, and dispensing with a written examination.In the Logic Programming unit, the computer-based examination is complemented by coursework designed to encourage originality as, along with writing a program, students have to write a description and present it as a mini-project, to test literacy and presentational skills (Callear, 1996).
The greatest advantages of computer-based examinations is that marking is done automatically and immediately.The extent of time-saving when using computer-based examinations is illustrated in Table 2.If the preparation time of objective tests is set against the marking time of an average Level 2 paper (say, 15 minutes per paper), it is clear from the table that the cut-off point is 20 students.After this, the time-saving becomes considerable.Furthermore, questions can be reused because a student does not have a paper to take away.Such considerations are very important in a system that is under increasing pressure.The other major advantage is that the marking is objective, with no personal element resulting from the lecturer's knowledge of a particular student.
The validity of computer-based examinations is often challenged, but issues such as the 'guessing factor' in multiple-choice tests have been completely refuted by the national examining boards, and the construction of a multiple-choice question paper with a realistic number of graded questions offers a valid alternative to written examinations (Callear, 1996).

Continuous assessment and in-class tests
The use of computer-based tests for continuous assessment has been the subject of a three-year study at Portsmouth, involving up to 90 Year 1 students on the Introductory Information Systems Analysis and Design unit of the HND in Computing (King, 1995;King, 1996).These students received three or four computer-based tests during one semester.The tests were generally run back-to-back, with one test being removed as another test was made available.Although the running of the computer examinations seemed to have generated few practical problems, the availability of resources, especially staff time, was a critical issue for continuous testing.As well as the creation of a question bank, staff workload for continuous computer-aided assessment (CAA) was generally found to be considerable.For open-access tests, there were up to twenty tasks, involving test control and student preparation and feedback, which needed to be co-ordinated at any one time.For summative tests, additional tasks could arise involving equipment availability, room booking, invigilation, and contingency planning.

Innovative assessment using CAA
Attempts were made while running the continuous computer-based test sequences to introduce some innovative aspects to the assessment process.
The tests were open-book, non-invigilated, not time-constrained, and with feedback to the students so that they could identify their areas of weakness.In later tests, the total scores were displayed immediately at the end of each test, supported by printed reports such as detailed exception reports on incorrect responses.The most formative aspects of the trial centred on the provision of valuable revision tests for self-assessment.The original tests were reissued with immediate feedback on each question and comprehensive assistance from one-line hints, contextual help pages, and tutorial pages which could be consulted before answering a question.

Future developments for computer-based testing
Future developments are likely to follow two routes: The increased use of computer-based examinations and summative tests for large groups Zakrewski (1996) from the University of Luton reports that 4,000 students were given computer-based examinations across eight different subject areas in the academic year 1995/6.In the Department of Information Science at Portsmouth, all students on first-or second-year units could be examined using computer-based examinations in about 20 hours.Savings in marking time in only two final-year units could exceed 80 hours for one lecturer.Results would also be available immediately to examination boards, a great advantage in the brief period available between Semesters 1 and 2.

The development of exploratory formative assessment as distinct from summative assessment
This may be done as informal self-assessment, with the results being entered by students into portfolios or log books, or test completion may be participatory only.Such tests would be feedback-driven and designed to maximize their usefulness to students in their learning.It would include printed feedback reports giving students full information on their own responses.The delivery of test questions would not be randomized, which would allow the use of graded questions, and would not disrupt the students' assessment of their own weak areas.These tests would contain or be incorporated with more overt forms of computer-based learning, and would be facilitated by a change of software to Question Mark Designer for Windows which would allow other applications to be run simultaneously with the test, such as ToolBook for interactive material, the Internet for tutorial notes, Java-or JavaScript-based simulations and animations, and email packages for contact with tutors or other students.The use of computer-based testing in this way would create a unified testing environment for the students, with summative tests being simply a more rigorous extension of the formative mode.

Facilitation of future developments
None of the above future developments is likely to be successful without being underpinned by three factors:

Adequate resourcing
Management commitment to computer-based testing in the form of adequate resourcing of the innovation is crucial to its long-term development.Commitment can be expressed in the form of support for the purchase and evaluation of suitable testing software and the maintenance of software site licences, increased funding for computer hardware, laboratories and staff development.Careful consideration should also be given to support the development of a new academic technician post of CAA Officer to take on the responsibility of link person between computer services and academic staff.A list of issues and tasks for consideration by a CAA Officer can be found in the checklist in the Appendix.Such a post would be vital for removing from academics the wide range of non-academic tasks associated with computer-based testing.

Improved technical support
To increase successfully the volume of testing, technical support for computer-based testing will need to be formalized and given a higher profile, and an error-free testing environment provided.Such a move will require management action, but work within technical support will be made considerably easier by liaison with just one key CAA Officer.

Assuring quality in the testing process
Reliability will be underpinned by sound operational procedures, and test validity can be ensured by the involvement of the course team in writing, grading, moderating and testing questions.Where there is a need for a very large question bank and a fast changeover to computer-based examinations, the only feasible approach may be to have the questions written by an outside agency but passed to the course team for moderation.

Conclusion
After some years developing expertise in successful CAA within Information Science, the department is now in a position to capitalize on this knowledge.However, future developments may be jeopardized if insufficient attention is paid to the creation of the assessment infrastructure which is vital for sound and high-quality testing.

Table I :
Different ways of assessing student abilities

Table 2 :
Time-savings using computer-