A STILE Project case study : the evaluation of a computer-based visual key for fossil identification

This paper presents an investigation of the effectiveness, for specimen identification, of a visual representation of a biological key. The example used in the investigation, the beetles database, is one of a number of resources developed under the STILE Project (Students' and Teachers' Integrated Learning Environment). This project uses hypermedia to provide greater opportunities for independent and flexible modes of learning both in a campus situation and for distance learning. The beetles database was constructed to aid final-year project students' in their identification of palaeo-ecological field specimens. The development of this database was a response to a perceived need to reduce time spent on the time-consuming skill of identification, and to focus students' efforts on the significance of their field data. Four third-year student undergraduates, two experienced and two inexperienced users of paper keys, were presented with a range of field specimens to identify using either the paper key or the STILE visual database. Our results show that the visual database was both the preferred way of operating and more effective than paper keys for all students.


Introduction
The STILE Project (Students' and Teachers' Integrated Learning Environment) is one of 76 projects set up under the UK Government's Teaching and Learning Technology Programme (TLTP) initiative sponsored by the British Higher Education Funding Councils (HEFCs).The STILE Project uses hypermedia to provide greater opportunities for independent and flexible modes of learning both in a campus situation and for distance learning.The approach is resource-based.STILE provides a mechanism for both tutors and learners to discover and access relevant resources when they need them, together with facilities that enable users readily to use and re-use existing materials, to integrate them together, and to add further materials of their own in a way that seems natural to them (see Ruggles et al, 1995).The result is not a closed and finished product, but a set of tools and services and a continually developing resource base.The effect is to ease the load on academic staff in maintaining and supporting student access to resources, and to enrich the set of resources available to both staff and students.
One such resource is an expanding database of beetles (insects of the coleopterous order).This paper presents an investigation of the effectiveness of a specific use of that database, as a visual representation of a biological key for the identification palaeo-ecological fragments.We would emphasize, however, that this type of resource could be developed, and is being developed, across a range of subject domains.

The problem
The final year of undergraduate courses in the UK is generally one of increasing specialization leading to an individual long study or dissertation.The reconstruction of past environments using palaeo-ecological field data is typical of such specialist studies.These studies have high academic credibility but are also resource-hungry in both tutor and student time, and as such are increasingly threatened by the need to reduce unit costs.This need for efficiency gains was the initial stimulus for the development of the beetles database.Specifically, our task was to ease the load on academic staff by providing easy access to resources for these and other students.
As part of our brief, we completed a task analysis from the collection of the field data to the final reconstruction of the past environment.From this we noted that while some 40% of the students' project time was spent on the core activities such as the collecting and preparation of field samples, analysis and report writing, 60% was spent on identification of field specimens.We felt that a disproportionately large amount of time was being consumed by this conceptually low-level task.It became apparent that there was scope not only for efficiency gains in staff time, and therefore costs, but that it might also prove possible to reduce the workload of students.This in turn would lead to benefits by releasing scarce laboratory time.

The resources
The task of identification is a subset of the overall skill of categorization.Categorization skills are the key to both the organization and retrieval of data.They allow the reduction of environmental data into a hierarchy of classes by discrimination, abstraction, generalization and organization of common elements or crucial aspects of stimuli.It has been argued that the organization of knowledge is the key to problem-solving and thus to the individual's successful adaptation to the world.This research emphasis on knowledge structures is underpinned by an extensive literature that has supported two main models of semantic memory: network models, based on the relationship between elements; and feature models, in which memory is organized by the identification and grouping of features (see Underwood, 1976;Solso, 1988;Anderson, 1995).
The two identification tools evaluated here, the paper-based key and the computer-based visual key, have different organizational structures as well as different modes of presentation.The former is strictly hierarchical, but the latter, although having a hierarchical overstructure, focuses on feature identification.

The paper-based key
There is only one current guide to British beetles (Joy, 1976), and this is the core text for specimen identification using a traditional paper-based key as shown in Figure 1.It has an overall hierarchical structure, in that within the hierarchy information is differentiated, and the properties at one level apply to all related units at the next lower level.For example, the description of the family Donacia supersedes and applies to the specific case of Donacia Cinerea.Membership of a category is not a matter of degree but is all or nothing.The use of such keys is difficult for the inexperienced.The terminology consists of a mixture of English and Latin names, and abbreviations are frequently employed.In addition, there is no use of visual information.Students at the start of their entomological studies may be excused for finding this key difficult to use.

The computer-based visual key
The beetles database has a topic net structure (Ruggles et al, 1995) which allows great flexibility in its use and focuses on the relationships between objects.For the specific purpose of specimen identification, a hierarchical structure was laid over the net with a focus on specific features or objects.The key starts at the level of families, and is further subdivided into orders.Within each order, each insect has a file consisting of one or more whole-specimen images, images of key body fragments, and information about the insect's food preferences and habitat (see FitzPatrick and Greenwood, 1996).

The trial
The development of the beetles database was a response to the perceived need to reduce time spent on the low-level skill of specimen identification, and to re-focus students' efforts on the significance of their field data.It was hypothesized that for moderately experienced users, use of the visual key would lead to faster identification of specimens.In addition, it was hypothesized that the use of the visual key, in removing the need to interpret paper keys, would provide a lower task entry-level which could be exploited by novice or inexperienced user.

Subjects
Four third-year undergraduates, two moderately experienced and two inexperienced workers in this field, were presented with a range of field specimens to identify, using either the paper key or the STILE visual database.The two experienced subjects had a working knowledge of paper keys and of computers.The inexperienced users had no knowledge of paper keys and very little experience of using this technology.

Materials
All the target specimens were elytra or wingcases -the most commonly found field fragment -from members of the Chrysomelidae family.Subjects inspected the specimens using a microscope with a video camera, linked to a screen.Identification was completed either using the visual key, presented on a Macintosh LCIII computer through the Netscape World Wide Web browser, or using a traditional paper-based key (Joy, 1976).

Procedure
Before the trial began, each subject was introduced to the identification tools.They were reassured that the trial was an assessment of the tools (that is, the computer database and paper key), and that they should at no time feel that this was a test of their own performance.Following this familiarization, the subjects were asked to identify four specimens.They were allowed up to 15 minutes with each specimen.To further aid our evaluation of the visual key, they.wereasked to provide a concurrent dialogue as they worked through the task.Again, reassurance of the focus of the research was provided.
A within-subject design with counterbalancing of factors was used.Each subject used the visual and the paper key for the identification of two of the specimens.The order of use of keys was counterbalanced, using an ABBA design, to reduce the effects of learning during the trial itself.Performance measures were task time and success rates.All subjects' comments were recorded.Post-trial interviews were conducted.

Results
Subjects attempted to complete the specimen identification in only 14 of the 16 trials.Two subjects, one experienced user (D) and one novice user (A), each identified one of the specimens correctly, and the second experienced user (C) produced a close match judged by our expert to be an acceptable answer; that is, user C had successfully negotiated the higher levels of the identification hierarchy, but had failed to distinguish between closely related samples.In all three of these cases the subjects were using the visual computerbased key.The second inexperienced user (B) failed to identify any of the specimens.
The use of the visual key was not without problems, however.Students seemed very disturbed by what they saw as disparities in the colour match between specimens and the images in the key.Secondly, two of the four students had navigational problems in recovering from an abortive trail.
No acceptable answers were recorded for subjects using the paper-based key.This was surprising, as Subject C was an experienced user of this tool although he had not worked with the Danacia family.One subject refused to complete the identification process when using the paper-based key.The following statements are typical of users' comments of the paper-based key: I reckon it's virtually impossible to tell from the key really.(Subject D) • / think I'm going to find this impossible.(Subject A)

Yeah, I'm going to have to give up again. I couldn't do it. (Subject B)
There was no discernible difference in the overall mean time on task between trials using the computer key (mean of 390.6 seconds) and the paper key (mean of 382.5 seconds).The shortest time was recorded for a non-computer trial when the subject had actively curtailed the trial (Table 1).
There was an interaction between type of tool used and level of experience, however.Figure 3 shows that the more experienced users spent longer with the paper-based key than the inexperienced users but that the position was reversed for the visual key.This small pilot study suggests that whereas the visual-based key enables the student to take part in the identification process, the paper-based key acts as a barrier.This barrier can be so great that the student effectively gives up.More experienced subjects were able to come to a conclusion more rapidly using the visual key with a success rate of 50% (two out of the four specimens).They were more willing to persevere with the paper-based key than the inexperienced users, and struggled to complete the task, hence the increase in identification time compared to the visual key.This perseverance was not met by success, however.Even the small shift from one family of beetles to another made the use of the paper key extremely difficult for Subject C.

Inexperienced
Post-trial interviews confirmed that the visual database was both the preferred and the more effective way of operating for all students.Subject A, inexperienced but with one successful identification using the visual key, admitted that she struggled with the paperbased key but found the visual key easy to use.She was most impressed by the low level of computer skills required to operate the software effectively.Subject B, also inexperienced, had found the paper-based key impossible but appreciated the facility to quickly browse through the visual key.Subject C, experienced, felt the visual key was potentially very useful and very easy to use but he had some problems because of his mild colour-blindness.Subject D, experienced, was convinced that the visual key would save him time.

Conclusion
While the small sample size for this pilot study necessarily means that the results should be viewed with caution, they are nevertheless encouraging.Our initial hypotheses were that moderately experienced users would identify the specimens more quickly using the visual key but at first sight the data does not support our hypothesis.Mean times for the use of the two keys are indistinguishable.This result can be explained, however, by the tendency of one inexperienced student to curtail sessions when working with the paper key and by other students' willingness to continue using the computer key after a preliminary identification had been made (Subjects A and C) to confirm a decision.Long search times with the paper-based key did not lead to success, and the shorter times were associated with a refusal to continue the trial.The efficiency gains that were apparent derive from the differential success rate for students when using the visual key compared to the paper key In addition, it was hypothesized that the use of the visual key would reduce the level of skills required by the novice or inexperienced researcher when first encountering the identification task.This was certainly confirmed.This initial simple measure of performance -correct identification -showed one novice user performing at a similar level to more experienced counterparts, and the second novice at least completing the task.

Figure
Figure I: A typical key entry

Figure 2 :
Figure 2: Identifying the order of the found fragment The user goes through three steps.

Figure 3 :
Figure 3: Mean time taken to complete an identification using the two keys by level of experience.