Evaluating EuroMET-a Web-based resource for meteorologists

Euro MET has been created to address the education and training needs of professional meteorologists and students in tertiary education throughout Europe and beyond Two module libraries have been developed for the WWW which are available in English, French, Spanish and German. The modules have been evaluated during their development and after completion. All the project partners used them in courses and training programmes within their institutions. This paper describes the formative and summative evaluation strategies used and some of the results obtained The efficacy of the evaluation approach adopted with respect to the size of the project is also discussed


Introduction
The education and training of meteorologists varies widely across Europe.General meteorology is taught in many universities, typically as part of other courses, for example, geography, while comprehensive courses in meteorology are taught in specialist departments at both undergraduate and postgraduate level.Due to the relatively small number of universities teaching meteorology at sufficiently advanced levels (approximately 5 per cent of all universities Europe-wide), National Meteorological Services (NMS) usually recruit employees at university science degree level and then offer training as is necessary.Another part of NMS education programmes is extension training, required to keep all employees up to date with the latest developments in meteorological science.NMS training varies from employees attending intensive courses running over several weeks, which can be very similar to a university module, to much less formal 'on-the-job' training.Regardless of the training offered, NMS, unlike universities, do not formally examine or assess their trainees.At present no distance-or open-learning courses in meteorology exist within Europe.
Computer-based learning resources could therefore be extremely useful within the European meteorological community.They could offer an alternative to those NMS with little or no training programmes of their own and enable NMS training centres and universities to share their educational material.Meteorology is also a subject that could benefit from being taught in a Computer Aided Learning (CAL) environment.Much meteorology concerns descriptions of atmospheric phenomena in three dimensions that vary in time, usually presented as complex mathematical equations.A CAL package offers the opportunity to visualize these descriptions as animations or models and simulations that can be explored and experimented with.The European Meteorological Education and Training (EuroMET) project therefore aims to address the education and training requirements of professional meteorologists and students in tertiary education throughout Europe by establishing a multimedia, networked-based, open and distance-learning service.

The EuroMET consortium
EuroMET is an EC-funded initiative with matched funding from the participating NMS.The consortium is comprised of twenty-two partners from fourteen European countries and the University of Quebec in Canada, divided into two groups, the developers who are responsible for producing the modules, and the evaluators who are responsible for assessing them.The developers are led by Met6o France (where the project co-ordinator is also based) and the University of Edinburgh.The evaluation is lead by EUMETSAT (the European Organization for the Exploitation of Meteorological Satellites) and the University of Reading.The thirty-month project commenced in June 1996 and is split into three key stages: the prototyping of minimal content modules, the development and evaluation of full versions of the modules, and finally, the implementation and evaluation of the modules into their teaching/training programmes by all partners.

The module libraries
Two module libraries have been developed; the subjects for which were selected in response to a European-wide needs analysis of the NMS.The two chosen were Numerical Weather Prediction and Satellite Meteorology.Both libraries are divided into nine chapters covering different sub-topics and contain around 70 modules each.Each module takes an average of 30-40 minutes to complete.The modular approach allows teachers at different locations to choose those modules that are directly relevant to the course they teach.
A common interface was developed so that all modules would look and feel the same.The interface was designed to be as simple as possible so that users could learn to use it as they went along.An online help facility is also available which explains all navigation buttons, symbols and icons used.The interface is language-independent, an important consideration for a multilingual project.Each module is constructed with an opening motivation page to introduce the topic and engage the user's interest.The next page lists the module's learning objectives, any pre-requisite modules the user should study first, any subsequent modules that follow on, and a list of the module's contents^ The following pages contain the teaching material, whilst the final page contains a summary of the material, drawing out the salient points of the topic.Users navigate through the modules by clicking on arrows on a button bar at the bottom of the screen, to progress page by page, forwards or backwards, or they can click on the 'module structure' button to go directly to any page of their choice.A 'course structure' button similarly allows the user to select any module.
This learning environment is supported by a glossary and keyword index, enabling the user to find easily those modules that contain material covering a term selected from the index.Communication facilities are also included, such as email to subject experts, two discussion groups and a chat area.Due to the number and variety of users, EuroMET is designed to be as flexible as possible with tools to create adapted packages and extra material.This tool-kit allows teachers to create their own online course from the EuroMET modules completely tailored for their needs.This addresses the concerns of Draper et al. (1996) that 'designers of CAL frequently do not design in detail the teaching within which its use will be embedded, that teachers make many local adaptations just as they do for all other teaching, and that even if designers did prescribe their use many teachers would still modify that use just as they do that of text-books' (p.26).

Formative evaluation
The EuroMET evaluation strategy was planned and executed by all the evaluation partners with advice from staff at the Institute of Educational Technology, the Open University and the National Physical Laboratory Usability Services (NPLUS).The modules were released onto the Web at three points during their development for evaluation.The evaluation team could therefore assess the first set of modules while the developers worked on the production of the second set.Enough time was incorporated before the second release date for the developers to act on the evaluators' report, so that both sets of modules could be modified to include the recommendations made.This was then repeated for the second and third module releases.In this way the modifications could be tested by the evaluation team, resulting in an iterative approach to the development work.The modules were evaluated at each institute by a number of users ranging from novices to subject-matter experts.Between 100 and 200 users evaluated modules in each phase.The aim of the formative evaluation was to assess the modules' ease of use, pedagogy (including ensuring scientific integrity) and possible utility in replacing conventional teaching.These issues were addressed in two questionnaires that were completed after using each module.One questionnaire addressed the broader issues of functionality and ease of use, while the other was directed towards the actual content of the module.The questionnaires consisted of yes/no and agreement scale questions, with comment areas in which subjects were encouraged to expand on their answers and make suggestions to the developers.

Results from the formative evaluation
Results from the questionnaires indicated that the modules were well-presented and easy to use.An observed session with a group of new users demonstrated that it took them only about ten minutes to become familiar with the navigation tools, despite not having seen the modules before or having received any instruction in their use.Regarding the module content, the vast majority of users preferred those modules.thatused as much interactive and visual material as possible to explain a subject and which also had as little text as practicable.A comment users frequently made was that they would rather read large amounts of text from a book than a computer screen.The evaluators therefore recommended that where textbooks were available to explain material in-depth, developers should attempt to produce highly visual and interactive modules.
A problem with the early modules was that of navigation; many users disliked the exploratory layout some modules adopted.Users often made comments that they felt 'lost' in these modules and were unsure if they had accessed all the material; one of the evaluators termed this type of module structure as 'spaghetti navigation'.In the light of these comments it was decided that all module navigation should be much more linear and that the user should be guided through the module.
The formative evaluation was extremely useful in producing modules that were suitable for their target users.As the evaluation was embedded into the development work it was easy for the developers to modify the modules according to the evaluators' recommendations and in turn for these modifications to be tested during the next evaluation phase.A marked improvement in the quality of the modules from the first to the third evaluation phase was seen.The evaluation also resulted in a more homogeneous feel to the modules.Although a common graphical interface had been developed there were still significant differences between modules.For example, no universally accepted standard notation exists in meteorology to describe various physical quantities, so different developers often used different symbols, and sometimes units, for the same term which can be confusing, particularly for someone new to meteorology.The evaluation helped to ensure that in the absence of any standard notation, at least that used in the modules was consistent throughout the EuroMET libraries.The formative evaluation has been discussed in more detail elsewhere -see Phelps and Reynolds (1999).

Summative evaluation
Unlike formative evaluation, summative evaluation is carried out after the design and development work is complete.The purpose of the summative evaluation is to measure a CAL package's effectiveness with regard to its stated aims and to assess its appropriateness for certain tasks and users.It therefore has to be undertaken in real learning settings and involve a larger number of subjects than formative evaluation (Chanier, 1996).Therefore, the modules have been used by all the project partners on courses at their institutions over a twelve-month period ending in October 1998.These uses include classroom demonstrations using the modules to illustrate a particular concept, full lecture replacement, using the modules as an electronic reference and independent study programmes for individual users.A range of evaluation methods is being employed to investigate these different uses and the effectiveness of the modules.To complement these evaluation tools, an evaluation site has been developed where the tools can be accessed, results from other partners' courses viewed and information on CAL implementation and evaluation strategies can be found.Although it is extremely difficult to isolate the effects of CAL from other learning resources (Draper et ah, 1996;Gunn, 1996), it is hoped that by collecting a range .ofdata from different learning environments it will be possible to ascertain whether EuroMET provides a useful and effective learning tool and to give us guidelines on how best to integrate it into both university and workplace settings.The evaluation methods used are now discussed individually in the context of the project.

Questionnaires
Two Web-based questionnaires were devised for users and teachers to complete after using the EuroMET modules.The questionnaires were designed to gather information on the users' background, how the modules were used for study, what technical and academic support was given, impressions of the course and so on.Results were collated automatically to gain an overview of each course and individual submissions were stored in a database for more detailed analysis.

Web-tracking
Javascript has been embedded into each EuroMET page.Each time a page is accessed, whether from the memory cache of the user's browser or either of the two EuroMET servers, the Javascript calls a CGI script on the server in Edinburgh.The CGI script then produces a tracking log recording such details as the host machine accessing EuroMET, the browser type and version, the date and time of access and the URL of the current and previous pages accessed in the browser session.This results in the collection of valuable data such as when and where users access modules, which modules they look at, which pages in those modules they access, how long they spend on each page, which language version they access, and so on.

Interviews and observation
Interviewing students and observing their interactions with the modules is a good way to obtain information that has not been taken into consideration when designing questionnaires.Open-ended discussion allows an evaluator to follow up comments made by students in order to understand fully what has taken place during a CAL session.

Assessment of users' knowledge
To investigate the pedagogical effectiveness of the modules, teachers have been encouraged to assess their students' learning from the EuroMET modules.Such assessments may be an end-of-year examination question, a forecasting exercise, or a presentation on a module subject to the user's peers.

Results of the summative evaluation
So far results have been obtained from 14 courses in 11 different institutions and results from a minimum of 12 more courses are expected in the final months of the project.These preliminary results reveal that the course has been widely used in a number of settings and by a range of learners.The modules have been used as extra material to accompany a lecture course, as material to replace a lecture, and for independent study by individuals.The data collected so far have shown that these three types of users have similar previous computing experience and that they all give EuroMET a comparable overall rating.It is also worth noting that all users received similar levels of supervision, both academic and technical.
The independent learners were all experienced forecasters aged between 35 and 45 years old.They were more positive about the learning benefits of EuroMET than the other two groups, as illustrated in Figures 1 and 2. Those who used the modules as extra material accompanying lectures were all students, ranging in age from under 25 to 45 years old and either in their fourth year or above, or higher-degree students.Although generally positive about their learning experience, they were less likely to think that using the modules had improved their skills/knowledge (Figure 1) than the independent learners.This is not surprising as generally these users were using the modules after a lecture to consolidate what had already been learnt rather than learning something completely new.Interestingly though, most of these users found that the modules helped them to visualize the material addressed (Figure 2).The third group, those users who had used the modules as a replacement to their lectures, showed a more mixed reaction to the modules.agree agree strongly By dividing the lecture replacement group into their respective courses a clearer pattern emerged.Three courses fell into this category, two received positive feedback, while the third provoked more negative responses.It was initially difficult to see why this might be so as one of the positively received courses was taught in the same institution in the same manner (during compulsory, timetabled computer sessions) as the negatively received one, though with a different set of students.However, on examining both sets of students' comments, it was seen that the modules used in the second course were found by most students to be 'a bit heavy going'.These modules had yet to be re-engineered and so were still very text-based and would have benefited from the addition of more figures and interactive elements.This result highlights the importance of the work done in the formative evaluation and illustrates that in this case it was not the implementation strategy that caused the disparity in users' opinions, but the quality of the modules studied.
The evaluation has shown that EuroMET has been successfully used on a number of different courses and in several different settings.The results also demonstrate the efficacy of the evaluation methods adopted.The formative evaluation resulted in modules that were easier to use and of higher quality.The evaluation Web site and online questionnaires developed for the summative evaluation have also been extremely useful.In order to be able to use the questionnaires correctly, partners had to register their course details on the evaluation site.Therefore, it was not only straightforward to collect valuable data from the questionnaires but also to monitor when each partner was running a course and when they should submit their completed questionnaires, an important consideration on such a large project.

Figure I :
Figure I: Users'responses when asked to what extent they agreed with the statements 'the modules help you to learn'and 'the modules have improved my professional skills/knowledge'.