Computer-aided assessment in statistics : the CAMPUS project

This paper describes the specification, features and implementation of computeraided assessment software designed primarily for conducting assessments in Statistics but equally applicable to other quantitative disciplines. The CAMPUS (Computer Aided Marking Program Using Spreadsheets) package has been developed in Microsoft Excel so as to provide a familiar computing environment for both assessors and students. The principal feature of CAMPUS is the facility for setting questions containing random elements (including random graphs), so that each student sits essentially the same questions but with different answers. This helps to eliminate plagiarism and allows a single test to be used many times, either for reassessment or additional practice. CAMPUS has been implemented in a higher-education context but could also be used at school level.


Introduction
The relentless drive for 'efficiency' in higher education, and the consequent increase in workloads, has given university teachers a compelling incentive to investigate alternative forms of assessment.Some forms of assessment with a clear educational value can no longer be entertained because of the burden placed on the teacher.An added concern is plagiarism, which anecdotal evidence would suggest is on the increase yet which is difficult to detect in large modules with more than one assessor.While computer-aided assessment (CAA) has an enthusiastic following, it is not clear to many teachers that it either reduces workloads or reduces the risk of cheating.In an ideal world, most teachers would prefer to give individual attention and personal feedback to each student when marking their work.In this sense CAA must be seen as second best and will therefore be used only if it is seen to offer significant benefits in terms of reduced workloads or increased validity.This paper describes the development of a spreadsheet-based CAA package for use in the teaching and learning of Statistics and other quantitative disciplines.The CAMPUS program (Computer-Aided Marking Program Using Spreadsheets) seeks to address the concerns of teachers regarding workload, validity and feedback when assessing students.

Requirements of a CAA package
A CAA package must be easy to use, both by the teacher when setting the assessment and by the student sitting it.If the CAA package requires a large investment of time in setting up the test, it will be used only by the enthusiast.Equally, the process of sitting the test must be straightforward, otherwise students will quickly seize upon software difficulties as the reason for their under-performance.The package should be able to accommodate a variety of question types, including multiple choice, text response and numerical answer.There should be a mechanism for giving informative feedback to students, not simply the number of 'right' and 'wrong' answers.An efficient and secure system for recording marks is required.Security devices are essential to guarantee the integrity of the test, ensuring that students can neither access it before it is sat nor alter their answers or marks after submitting it for assessment.
The nature of Statistics as a discipline imposes additional requirements on a CAA package.Many areas of Statistics involve a graphical approach, so the ability to incorporate graphs and charts into questions is essential.Statistical theory is quite mathematical, so there is a need to be able to handle symbolic notation.Most statistical problems involve a mixture of calculation and interpretation.While the calculation element lends itself to questions with a fixed numerical answer, assessing interpretation is more difficult.Using text response is almost impossible in most cases since there are many different forms of words that have the same essential meaning.Multiple-choice items are not an entirely satisfactory solution since they may give the student ideas they may not otherwise have had.
Most teachers will already possess a battery of assessment items that they have accumulated during their career.Some of these may be paper-based, but many will be stored in a variety of electronic formats.For example, I have maintained a bank of some 400 tutorial questions stored in text format on a Unix computer.It is clearly desirable that these accumulated resources should be easily transferable into any CAA package the teacher chooses to use.

Random questions
Software already exists which offers most if not all of the desirable features outlined above.For example, Question Mark Designer (QMD) is in widespread use within UK highereducation institutions.What motivated the CAMPUS project was the desire to able to set random questions.Software such as QMD offers a facility to select a certain number of questions at random from a large library, but it does not allow individual questions to contain random parameters.To clarify the distinction, consider a trivial item designed to test students' ability to carry out simple addition.A large library of questions could be prepared, as in Table 1.The alternative is to prepare a single question, as in Table 2, where x and y are numbers that are randomly generated each time the question is loaded.

Random question What is [x] + [y] ? Ans=[x] + \y] | j a bk 2: Random question
This has several advantages: 1.All students will tackle different questions, eliminating any possibility of copying answers.2. Provided the random parameters are constrained to lie within reasonable limits, all students will tackle questions of virtually equal difficulty, which fairness demands.3.There is almost infinite opportunity for drill and practice, which many students need.4. Statistics is an applied subject and the vast majority of Statistics teachers would endeavour to set assignments and problems in a real-world context.Statistics is taught to students in many disciplines: biology, geography, economics, psychology, to name but a few.Finding different question scenarios for each of these diverse fields of study is very time-consuming.The task of establishing a large library of questions of similar difficulty for each application area would be enormous.5.With such a trivial example, there is little difference in the effort required by a lecturer in setting up the test.However, in more complex problems, even the task of computing the answers would be very tedious (let alone the risk of occasionally getting an answer wrong!).

Microsoft Excel
The desire to incorporate random elements, including random graphs, leads naturally to a spreadsheet-based system.Microsoft Excel is the spreadsheet package supported by Coventry University and is widely used elsewhere.An increasing number of students have home computers on which Excel is installed.As a basis for a CAA package, Excel has many advantages: • it includes a large range of statistical functions; • it incorporates a random number facility; • the Chart Wizard offers a wide range of statistical charts and graphs; • charts, graphs and calculated statistics can be linked dynamically to the source data; • there is a limited use of symbolic notation and an ability to import Microsoft Equation Editor objects; • the Import Wizard allows a lecturer to import questions stored in a variety of electronic formats; • the software environment is familiar to the students and will not therefore distract from the test; • the software is familiar to staff who will therefore require little training to use the CAA package; • the user-interface can be controlled (and secured) using Visual Basic macros.
Simonite et al (1998) have described an Excel-based system for setting personalized coursework assignments for students using a randomization method.However, the marking of the assignments is still carried out by hand.The CAMPUS program seeks to automate the whole process of setting and marking such assignments.The use of Excel as an authoring tool is not.initself novel.The CLUES project at the University of Aberdeen (Tidball, 1995) has led to the development of Excel-based courseware in many different disciplines.Similarly, the DISCUS project (Hunt, 1996) at Coventry University has produced interactive materials for teaching Statistics.|^^| Your answer here 3

The CAMPUS program
3) If the mean and median of a set of data are the same, which of the following statements must be true?A All the data values are the same B The distribution is Normal C The mode is also the same as the mean D The distribution is symmetrical E The standard deviation equals the interquartile range F None of these ^^^H Your answer here 2 completed the selection of questions, on the click of a button both questions and answers are automatically copied from QUEST and pasted into the TEST template.The answers are immediately hidden from view, and the TEST file is then saved under a different filename specified by the teacher.
A student sitting the test simply loads this file into Excel and follows the on-screen instructions (see Figure 1).Visual Basic macros within the test file ensure that any random elements within the test -in both questions and answers -are recalculated on loading, so that each student's test is different.Having completed the test, the student clicks a button to 'Submit for Marking'.A macro then checks the student's responses against the (hidden) given answers and compiles a detailed feedback sheet for the student (see Figure 2).Two copies of the student's responses and feedback sheet are automatically sent to the local printer.The teacher may also have opted for them to be saved electronically to a secret network directory.

Question construction
Five different types of question are supported by CAMPUS.

(i) Multiple-choice
These are the simplest to construct.Figure 3 shows the layout of a typical question.
The question is constructed on the first 17 columns of a standard spreadsheet.On the first row of the question, Columns 1-4 are for reference information.Each of the four lines in the body of the question is actually entered into a single cell in Column 5 -in Excel, cells automatically expand to accommodate up to 256 characters.The question text could be copied in from a word-processed file, provided that a carriage return is inserted at the end of each line -if not, the whole question will be pasted into a single cell.
The lines immediately following the question contain the possible answers -a maximum of nine choices can be given.In Column 5 are the labels (A, B, C, etc.) while Column 6 contains the answers that students are expected to give -both correct and incorrect.In Column 12 a feedback narrative is entered corresponding to each answer.This narrative

(ii) Numerical answer
The structure of numerical-answer questions is almost identical to that of multiple-choice questions.The only modification to the layout in Figure 3 is that the letters A, B, C and D, labelling the four choices, are replaced by numbers indicating the maximum acceptable arithmetic error (either way) in the student's answer.For probabilities this might be 0.0005.If omitted, these tolerances are assumed to be zero.Obviously, the four answers are not now displayed to students sitting the test.However, if a student enters one of the foreseen wrong answers he or she still gets the corresponding feedback narrative.Once again, up to nine possible answers can be entered.One minor restriction is that the correct answer must be listed first.
(iii) Multi-part It is quite common for Statistics questions to comprise several parts, all based on the same real (or at least realistic) context.Suppose that our example question has an additional part: 'What is the probability of at least one of its ships running aground this year?'.The revised layout is shown in Figure 4.
Each part is effectively a separate question.The only link is via the reference numbers 5.1, 5.2, and so on.Part (a) must be incorporated into the body of the introduction.When constructing a test, the setter may omit some of the parts, provided that the labelling is edited accordingly.

(iv) Random element
The ability to set random questions is probably the key feature of CAMPUS.These are the most difficult to set and require a basic grasp of Excel functions and formula construction.
Figure 5 shows a modification of our initial example to incorporate a random element.
In the original question students had to calculate the probability of two ships running aground.Now, each student calculates the probability of x ships running aground, where x is randomly chosen.The critical line is the fourth line of the question.In Column 5 the setter starts typing the question as normal.At the point where the setter would have typed '2 ships ...' he or she presses Return.In the next cell, Column 10 here, an Excel formula is entered.It seems reasonable for x to be 0, 1, 2, 3 or 4, so the appropriate formula is =INT(5*RAND()).Pressing the F9 (recalculate) key a few times acts as a useful check that the formula is operating correctly.
It remains to sort out the answers.The first answer should be Pr(x) using mean 1, the function for which is POISSON(x,l,FALSE).The location of x must be specified relatively using RC (Row and Column) notation.In the spreadsheet shown, this would be R meaning back 1 row and forward 4 columns.
So the final formula would be: Similarly, the second answer would be: the third: and the fourth: Currently, CAMPUS does not allow random feedback narratives, so the fourth feedback narrative would have to say, albeit rather vaguely, 'Wrong: you calculated at least not exactly'.
(v) Graphic element Provided any graph, chart or picture is embedded into the cells containing the body of a question, it will be transferred across to the test paper in the normal way.If the chart contains random elements, the source data must also be stored in the body of the question.
Since embedded charts actually sit on top of the cells, the source data can be hidden in the cells behind the chart if necessary.

Uses of CAMPUS
At Coventry University, the CAMPUS program has been used to conduct three different kinds of assessment.
(i) Supervised summative assessments Here, students all sit the test at the same time in a networked computer laboratory under the supervision of a member of the teaching staff.The teacher responsible for the assessment will have previously installed the test file onto the network, which students now load into Excel.On completion of the test, the student feedback sheets are retrieved immediately from the local printer, one copy being retained by the teacher and the other presented to the student.Where there is any possibility of fraud, perhaps because the local printer is located in another room or some other insecure location, the teacher may crosscheck the printouts against the saved electronic copy of each student's work.
(ii) Unsupervised summative assessments The system has been used for marking unsupervised homework exercises, despite the obvious concerns regarding cheating.A student may have attempted the test several times, thereby benefiting from the feedback, or sought assistance from others who have already obtained the correct answers.Surprisingly, experience suggests that most students are content to settle for a mark of 50 to 60 per cent at the first attempt, perhaps regarding repeat attempts as a mild form of dishonesty.From an educational perspective, one would prefer them to take time to locate and correct their errors.Provided the homework does not carry any great weight in the overall assessment, the fact that a student has gained an inflated mark of 90 per cent is considered a small price to pay for the reduction in the marking burden.Colleagues who find this unsatisfactory simply ask for the student's handwritten solutions to be submitted with the CAMPUS printout, as an additional check that the work has been done.This also enables the teacher to identify the cause of any 'unforeseen error' feedback given by CAMPUS.Even so, the time taken to check homework in this way is a small fraction of the time taken previously to mark it and give relevant feedback by hand.

(iii) Formative self-assessments
The teacher can use CAMPUS to provide students with random revision tests that can be sat time and time again, until the student achieves an adequate level of proficiency.Since every time the test is loaded it contains different values for random elements, in this mode of operation students are allowed to save their test with its current values so that they can re-sit exactly the same test.Also in this mode of test, the electronic storage of a student's responses is disabled, to avoid unnecessary file storage -the student simply retrieves the feedback sheet from the local printer.

Security
It is the responsibility of the setter to ensure that the location of the test file is not made public prior to the appointed time for the test to be sat.When a student loads the test, several security devices come into operation: • all Excel menus and toolbars are disabled and removed from view; • the answers and feedback narratives are hidden from view; • the test worksheet is locked, protected by a password supplied by the setter, so that only those cells reserved for the student's responses are accessible; • certain keys are disabled, most notably the '=' key, which might otherwise allow students to use Excel as a (very powerful) calculator; • in a Windows 3.1 environment, the Excel window is frozen so that students cannot switch to other software applications (for example, electronic mail), although unfortunately it appears to be impossible to achieve this in a Windows 95 environment; • the cells for a student's responses are formatted with a black font on a red background, which is remarkably difficult to read (even from an adjacent desk!); • once students submit the test for marking, their mark/feedback sheet is sent to the local printer and stored in a secure directory on the network.The permissions are set on this directory to allow students to create new files but not to read, write, view or erase them.
As an additional level of security, an Excel facility is used which only allows the file to be opened using the setter's password.
One weakness of CAMPUS is that although each student is required to enter a name before the test is marked, no steps are taken to confirm his/her identity.In a supervised test, it is left to the invigilator to check the identities of students sitting the test.

Conclusion
As more experience of running CAMPUS is gained, additional minor modifications are being made.However, the underlying simplicity of the package is its principal attraction, and no further major developments are planned.Work is ongoing to extend the existing database of questions.

Figure I :
Figure I: Extract from a typical test Figure 4: Layout of a muki-part question 12 3 4 Poison General 5

Table I :
Library of

VERY ELEMENTARY STATISTICS TEST Use TAB to move between answer boxes. Marks for each question are indicated in red.
CAMPUS consists of two Microsoft Excel 5 workbooks, QUEST and TEST.The teacher uses QUEST to construct a test paper by either selecting questions from the existing database, or by entering additional questions they have devised.Questions in the database are classified by topic (for example, NORMAL) and application area (for example, BUSINESS).TEST is a skeleton (template) test sheet which has already been formatted and equipped with various security features (described later).Once a teacher has