An evaluation of a teaching package constructed using a Web-based lecture recorder

This paper describes an evaluation of a teaching package used to replace lectures in two closely related university courses on Discrete Mathematics. The package was developed using Audiograph, a Web-based lecture recorder developed at the University of Surrey. Two groups of subjects were studied: a group of undergraduates, mostly fresh from schools, and a group of postgraduates, mostly with post-university work experience. Although the postgraduates with their greater maturity and experience were significantly more positive in their appraisal than the undergraduates, both groups agreed on the beneficial aspects of being able to work at one's own time and pace, and being able to repeat material at will. It is clear, however, that, in the context investigated, where the lecturer was readily available, such a package can never supplant a human teacher, and that considerable effort needs to be expended in order to integrate the package into a rich learning environment.

Packages consist of groups of 'slides' (Java applets) together with an HTML wrapper, and may be run from any Java-compatible Web browser.The learner uses the browser to invoke the applet associated with a slide, and the lecturer's hand-written presentation unfolds against an initial 'backdrop', accompanied by an audio commentary, again provided by the lecturer,.A typical slide, towards the end of its 'run', is illustrated in Figure 1.A scroll bar enables the learner to go backwards/forwards through the material on the slide, and mouse-clicks start and

•'•f
Example: He's playing^e If he's playing tennis QieMl fail his examsT herefore he'U fail his exams I • Rult of dttachmtnt fmtthod of affirming", Ari) I Figure I:A typical slide from a package developed using Audiograph.On invoking the associated applet the window (slide) is initialized with the printed background.Clicking on the slide causes the material to unfold: the lecturer's commentary is heard as various parts of the slide are highlighted, and hand-written additions appear.The user may skip, and repeat, material by using the scroll bar, and may invoke other slides by means of the 'hot words 'underneath.stop the unfolding of the material.Navigation through the group of slides is by the usual hypertext methods.The lecturer may enhance the presentation by the use of highlighting, by erasing portions of the screen, and by pausing in the delivery.Other Web-based resources (such as glossaries, simulations and extra examples) may be added as the lecturer thinks fit, though of course the provision of these extra resources adds to development time.
The evaluation reported in this paper centres on two groups of students taking courses in Discrete Mathematics in the Department of Electronic and Electrical Engineering at the University of Surrey.The delivery mechanism for each course was a teaching package developed using Audiograph.Students followed the course, each in his/her own time and at his/her own pace, using either a University laboratory workstation over the intranet, or the student's own PC, using a CD-ROM.In the latter case, it was possible for students to work in small groups, and those who availed themselves of this opportunity found it very beneficial.Regular conventional example classes and tutorials were provided by the lecturer, but attendance at these was rather variable, as discussed below in section 2.1.
The groups represented two distinct populations of subjects.One comprised first-year undergraduate students, mostly straight from school, and the other of postgraduate students on an MSc course, of whom all but one were over 25 years of age, and most were on secondment from full-time employment.As we shall see (in Section 3), the two groups differed significantly in their overall perception of the software, and of the course, and of the use of self-paced computer-based teaching material in general.They were, however, in overall agreement on the advantages of being able to work in their own time and at their own pace, and of being able to repeat material.In addition, they were in agreement that the package could not entirely supplant the teacher: not being able to ask questions as the material unfolded was seen as being a disadvantage by all but one of the respondents.

Background to the evaluation
A piece of software intended for the development of teaching packages may be evaluated with respect to several points of view: that of the management, where the evaluation questions are to do with cost and efficiency; that of the developer, where the issues concern ease of use in developing a package; and that of the end-user, where learning outcomes rather than general issues of ease of use are the prime concern.I deal here primarily with the latter viewpoint.As argued in Draper (1997) and Gunn (1997), such packages can be evaluated realistically only within the context of an actual course, using students registered on that course.Evaluations carried out in the real world, such as the one described here, frequently suggest ways in which the teaching package might complement other available resources (such as textbooks, and, most importantly, human teachers) so as to provide a rich learning environment within the context of the given course content and student population.Draper (1997) refers to such evaluations as 'integrative'.

2.1.The courses
The evaluation described here is concerned with the first 'real-world' use made of Audiograph.This was to construct a teaching package to replace the lectures associated with the Mathematics of Computing course, as given to all first-year undergraduates in the Department of Electronic and Electrical Engineering.This course, consisting as it does of very rigorous mathematics, is traditionally thought to be very difficult for engineering students.Not only is it very unlikely that such students will ever have come across rigorous definitions and proofs in their earlier scholastic careers, but also the need for such definitions and proofs in the future is not immediately obvious; that is, their motivation may be limited to the immediate goal of passing the examination.In addition, a suitably modified version of the package was used to replace lectures on the Discrete Mathematics course given to 11 postgraduates studying for an MSc.
Students on both courses were provided with an explanation as to how the course was going to be presented, together with printed notes consisting of the headings which appeared on each slide as it was initialized, which they were encouraged to augment manually as they ran the software and watched each slide unfold.The lecturer provided two tutorials a week for each course, the tutorials functioning mainly as problem classes, with the lecturer going over examples.The lecturer's perception is that nearly all the postgraduates attended nearly all the tutorials, whereas undergraduate attendance depended heavily on whether or not course tests were imminent.

2.2.The evaluation method
The evaluation centred on the analysis of data from questionnaires.These questionnaires were designed in the light of preliminary interviews with a handful of undergraduate students and with the lecturer concerned, and in the light of the investigator's own experiences in interacting with the teaching package.They consisted of both closed and open questions (requiring answers to be chosen from a list of provided alternatives, or inviting students to make their own comments).
Distribution of the questionnaires differed for the two groups.In the case of the undergraduates, questionnaires were sent out via internal mail on week 10 of a 12-week course to each registered student.Twenty-five of the 104 students on the course responded.It is impossible to know how representative of their population the undergraduate respondents were, although, given the method of distribution, the response rate of 24 per cent might be considered respectable.In the case of the postgraduates, the investigator distributed the questionnaires at the start of a tutorial, and again towards the end of the course, and the students completed them anonymously there and then.All 11 of the students registered on this course supplied responses.

Evaluation results
In general, the postgraduates, with their greater experience (specifically of university educa-tion), were more positive in their appraisals, but there was broad agreement across the two groups on the main advantage (the ability to run and repeat the material at one's own pace and time), and the major disadvantage (the inability to ask the lecturer questions as the material unfolds).

Overall perceptions of the teaching package and the course
Since the main aim of the package is to provide a lecture replacement, the first question asked of respondents was: Overall, how do you feel about using the software compared with attending conventional lectures?The results are show in Figure 2, which illustrates the difference between the undergraduate and postgraduate responses.The postgraduates were significantly more likely to give a positive response (5/9 versus 4/25, using a test for proportions gives a test statistic of z = 2.3, significant at the 5 per cent level).However, as we shall see, the apparently negative perceptions of the undergraduates in Figure 2 do not paint the full picture.The same qualitative difference between undergraduates and postgraduates can be seen (Figure 3) in the responses to the question: How easy/difficult did you find the course?We see that postgraduates were significantly more likely to find the course 'quite easy' than undergraduates (5/11 versus 2/25, using a test for proportions gives test statistic z = 2.6, significant at the 1 per cent level).Given their responses to the questions above, it is not surprising that the postgraduates were again more positive in their responses (Figure 4) to the question: How well do you now feel that you understand the course?It should be noted from Figure 4, that, along with 100 per cent of the postgraduates, 56 per cent of the undergraduates felt they mostly understood the course at this point towards its end, despite their perception of its difficulty, as in Figure 3.One might reasonably deduce that the teaching package fulfilled its primary purpose; the majority of the respondents over the two groups felt they understood the material.
We have seen that the postgraduates on the whole thought using the package was somewhat better than attending conventional lectures; were as likely to say that the course was 'quite easy' as 'quite difficult'; and all felt they mostly understood the course.The undergraduates on the whole found the teaching package to be somewhat or much worse than attending a conventional lecture; found the course quite, or very, difficult; but mostly understood it.
Having considered the respondents' general perceptions of the software and the course, I turn my attention in the next two sections to their perceptions of the advantages/disadvantages of using a lecture-emulation package as the main delivery mechanism of a course.Again we see that, in general, the postgraduates are significantly more positive in the results of their appraisal than the undergraduates, but that there is broad agreement between the two groups as to what constitutes the major advantages and disadvantage.
no ben.D some ben • great ben

I 2
No benefit Some benefit • Great benefit

Some advantages of the teaching package
Following the preliminary interviews with the lecturer concerned and a handful of students, some possible advantages of the package were identified.They were:-1.being able to work at a time which suits the user; 2. being able to repeat slides and parts of slides; 3. being able to work at the user's own pace; 4. being able to navigate easily round the course; 5. seeing the material unfold; 6. seeing the emphasis the lecturer puts on certain objects (by highlighting, or erasing, or by the moving finger icon).
Items 1 to 3 above were identified as advantages the package might have over lectures, 4 to 6 as advantages it might have over reading a textbook.In each case, students were asked to mark whether the item was perceived as being of no benefit, of some benefit or of great benefit.Figure 5a gives the responses of the undergraduates, and Figure 5b those of the postgraduates, both as numbers of respondents, rather than percentages.
We see that the postgraduates are overall more positive than the undergraduates, with item 5 (seeing the material unfold) being seen as of least benefit by both groups.At least two thirds of the respondents in each of the groups find each of the other items beneficial.
The open comments give some explanation as to why items 1 and 3 were not seen as being universally beneficial by the undergraduate respondents.As regards being able to set one's own learning pace, certain comments indicated that some, at least, of the learners wanted an indication of what that pace should be.And as for being able to work at one's own time, it was clear that this was not entirely true for that majority of respondents who were dependent on using the workstations (there were constraints on laboratory access).The course's generally sequential nature, due to the structure of its mathematical content, meant that ease of navigation (item 4) was not the asset it might be in other courses.As for item 5 (the dynamic unfolding of the material), the package perhaps did not fully exploit this facility.For example, slides illustrating proof were often concerned with the static product of a proof process (such slides were initialized with the proof, and when run, the commentary explained the proof step by step in a top-down manner) rather than with the dynamic thought processes (often non-linear and backtracking) by which the proof was produced.It is not clear why.item 6 (seeing parts of the material emphasized dynamically) was viewed as being beneficial by all the postgraduates, and of no benefit by 40 per cent of the undergraduates.
The context in which the teaching package was used played a part in how it was perceived.I have already mentioned that students who depended on laboratory machines suffered access problems; on the other hand, students who used the package on a CD-ROM on their own machine were free to follow the material in a group.Two of the undergraduates and five of the postgraduates worked in this way, and all found it beneficial.
Asking if there were any other advantages the software package had over attending a lecture course, or over reading a textbook or watching a video, elicited comments indicating, for example, that it was useful to see problems worked out step by step with explanations (an undergraduate comment), and that since the software package was more dynamic than a textbook, it held one's attention more (a postgraduate comment).

Some disadvantages of the teaching package
Respondents were asked to comment on possible disadvantages the package might have over conventional lectures.These disadvantages were identified by informal interviews as: 1. not being able to ask the lecturer questions as the material is being presented; 2. not being able to ask fellow students questions as the material is being presented; 3. not being able to see the lecturer as the material unfolds; 4. having to discipline oneself to work on the computer; 5. having to spend too much time on the computer.
The results are as in Figures 6a (undergraduates) and 6b (postgraduates).
As in section 3.2, we see significant differences between the undergraduate and postgraduate responses.Postgraduates are significantly less likely, at the 1 per cent level using a test for proportions, to see items 4 and 5 (having to discipline oneself to work, and having to spend too much time at a computer) as being disadvantageous.The difference in responses to item 2 (not being able to ask fellow-students questions as the material unfolds) might be ascribed to the fact that this situation did not arise with the five of the 11 postgraduates who worked in a group with a CD-ROM.
There is a startling similarity between Figures 6a and 6b in that both undergraduates and postgraduates overwhelmingly found disadvantageous the inability to ask the lecturer questions as the material unfolded.The investigator wondered whether this disadvantage might be more perceived than real, especially in the case of the undergraduates.After all, how feasible is it to ask questions in a lecture of upwards of 100 attendees?How likely is it that an attendee is following the lecture closely enough (and note that we are concerned with a course full of new concepts and ideas) to be able to formulate a meaningful question on the spot?D not a dis.What can certainly be inferred from this reaction is that the teaching package cannot entirely supplant the human teacher.Indeed, many respondent comments (both undergraduate and postgraduate) indicated that the package might complement -but emphatically not replace -the input of a human teacher.

How the teaching package might be improved
Given that this paper reports an evaluation of the first time that a package developed using Audiograph had been used in a university teaching setting, it is not surprising that there were glitches in the software and shortcomings in the supplementary material provided.Both undergraduate and postgraduate students requested more examples and alternative explanations.It would, of course, be easy to-supply these by way of hypertext links, but it has to be noted that this would increase, perhaps by a considerable amount, the development time for the package.
As regards shortcomings in the package software, concern was expressed regarding the slow initialization and running of the package, memory problems resulting in crashes, and problems with the audio.Supporting software was required to run the package, viz: Windows 95, an Acrobat reader and a Java-compatible browser, and these contributed their own problems.In total, 19 of the 36 respondents mentioned hardware or low-level software problems.Problems seem to be inevitable in the first serious use of a piece of software: one assumes that some will be easy to address, while others (especially those associated with the supporting software) will never be eradicated.In either case, slowness and unreliability do nothing to enhance the learner's positive experiences with running the package.

Discussion
I well remember in my undergraduate days sitting in a large stuffy hall, accompanied by upwards of a hundred others, indulging in an activity which was at the same time both frenetic and boring: attending a conventional lecture (in mathematics or physics).The activity was frenetic because of the speed of scribbling required in order to capture the precious words of the lecturer as the material was delivered; it was boring because there was no way that anybody could keep up with the concepts and arguments as they gushed forth.The intellectual engagement of the learner at the time was minimal.It was quite clear to me that the point of a lecture was to define the syllabus and amass material: all the serious learning took place elsewhere.This was 20 years ago, and possibly the situation has changed, at least with the increased provision of course handouts in the light of easy access to word processors, laser printers and photocopiers.None the less, a conventional lecture, in which one teacher in real time tries to satisfy the learning needs of a hundred or more learners, has serious deficiencies.Packages developed using the lecture emulation software of Audiograph seek to address these deficiencies by allowing learners to follow the presentation at their own pace, pausing and repeating as necessary, and using hypertext links to supplement material where desirable.From the lecturer's point of view, the core package, disregarding any supplementary materials, is almost as easy and quick to construct as the materials (slides/notes) for a conventional lecture.This is the theory.As to the practice, this paper reports on an investigation of two quite different groups of students attending two related courses where conventional lectures had been replaced by a teaching package developed using Audiograph.The undergraduates tended to work alone, on networked workstations, and to attend tutorials only when a course test was imminent.The postgraduates were more likely than the undergraduates to work in small groups running the package on a PC, and, in general, attended all the tutorials provided.The course was at a level of abstraction and rigour that certainly the undergraduates would be extremely unlikely to have met before.
As we have seen, the postgraduates were more positive than the undergraduates in their appraisals.They were more likely to find the package better than conventional lectures (of which, presumably, they had far more experience than the undergraduates); more likely to find the course quite easy, and less likely to chafe at having to exercise self-discipline in order to sit down at a computer and run the package.Both groups found almost universally beneficial the ability to work in one's own time and to repeat material.And over two thirds of all the respondents claimed that they had 'mostly understood' the course, that is, the package had achieved its primary objective.
The major disadvantage was seen as the inability to ask questions of the lecturer as the material unfolded.But -witness the (real-life) scenario at the beginning of this section -1 would claim that meaningful questions from the floor rarely form a big feature of a conceptually dense lecture to a large audience.Perhaps the strong negative reaction to item 1 in Figures 6a and 6b simply signals the fact that the learners wanted more human support, and more immediate human support.Consideration of this response and of comments made by the respondents concerning the interrelationship between the package and the human teacher, together with comments regarding the difficulty in knowing how well one is doing and whether or not one is keeping up, and the need for more examples and alternative explanations, leads me to believe that a teacher should think long and hard as to how the teaching package is to be supported and integrated into a course.Should a printed schedule be provided?How about self-assessment questions; a 'help-desk' accessible by email; regular tests (especially for the undergraduates)?It seems clear that the package on its own, though representing an improvement on the lecture scenario sketched at the beginning of this section, cannot supplant the human teacher.Rather, by taking over the rather mundane work of mass delivery, the package frees the teacher to concentrate on developing enhancement activities and material, and providing support for those in most need.
What I have been describing and discussing is the first use of a lecture-emulating package in a local environment, that is, an environment in which the lecturer is available.It could easily supply conventional lectures if the lecturer so wished.There are reasons for supposing that such a package might be extremely useful in a distance-learning environment (supported, perhaps, by an electronic 'help-desk').Plans are afoot at the University of Surrey for using Audiograph to provide a lecture-emulation package on CD-ROM for overseas students seeking direct entry into the second year of a degree course, in order to ensure that they have the necessary prerequisite knowledge.

Figure 5b :
Figure 5b: Postgraduate perceptions of the advantages of using the teaching package (see section 3.2 for descriptions of items 1-6)

Figure 6a :
Figure 6a: Perceived disadvantages in using the package (undergraduates) (see section 3.3 for descriptions of items 1-5)