Evaluating learning and teaching technologies in further education

With the current emphasis on quality assessment and the role of evaluation in quality assessment, it is likely that teachers in post-compulsory education will increasingly be expected to evaluate their teaching, especially when making changes to their teaching methods. In Further Education (FE), there have been a number of developments to foster the use of Information and Learning Technologies (ILT), following the publication of the Higginson Report in 1996. However, there is some evidence that the adoption of ILT has been patchy.


Introduction
There is currently an unprecedented interest in the use of technologies for supporting teaching and learning. In post-compulsory education, the current Government's commitment to increasing access to Lifelong Learning is expressed through a number of initiatives that also affect the further education (FE) sector. For example, in The Learning Age: A Renaissance for a New Britain (Stationery Office, 1998) the government outlines its proposal to expand the scale, scope and nature of both further and higher education. The Learning Age follows a number of such government papers that emphasize the importance of Information and Communication Technologies (ICTs) or Information and Learning Technologies (ILTs) in FE and HE.
For several years, HE and FE have been exploring the use of ICTs as a way of serving an expanding student population and allowing more flexibility in study patterns. This has resulted in the growth of flexible and distance education for lifelong learning and in the increased use of new technologies for supporting teaching and learning. In the FE sector, the report of the Learning and Technology committee of the FEFC (Higginson, 1996) recommended a series of strategic developments to foster the use of information and communication technologies across the sector. The first response to Higginson's recommendations was the funding by FEFC of a national staff development programme, QUILT (Quality in Information and Learning Technologies) managed by the Further Education Development Agency (FEDA) in conjunction with the British Educational Communications and Technology Agency (Becta).
In this context, evaluation is crucial. It offers an opportunity to investigate what is actually happening with the use of learning technologies and how the reality of the use maps onto the potential. This paper begins by briefly reviewing three recent approaches to evaluating ILT in HE. It then reports on a project, funded by FEDA, to develop a tool for FE colleges to evaluate their use of teaching and learning technologies: the project was entitled Learning with ILT. The paper discusses how the team that worked on this project approached the task, the process of developing the tool, issues that have arisen from the project and the implications for evaluating teaching and learning technologies in FE.

Models of evaluation of ILT
This section briefly considers three recent approaches to evaluating ILT: two approaches for evaluating ILT in HE and the development of an evaluation toolkit for HE lecturers and then draws out some principles which were used in the Learning with ILT project.

CIAO!
The CIAO! framework was developed at the Open University (OU) for evaluating the use of ILT in OU courses (Jones, Scanlon, Tosunoglu, Butcher, Greenberg, Murphy and Ross, 1996). It has three main dimensions: context, interactions and outcomes, and is intended to provide a variety of approaches to be considered for particular evaluations. The importance of considering the whole learning experience is reflected in the dimension of context which includes, for example, how the ILT component fits within the course; the context of use (at home, in a classroom, by individuals or groups) and how it is supported by human teaching. One important aspect of context is the designer's rationale in introducing the technology. Knowing the rationale enables us to decide on appropriate and achievable evaluation questions. For example, in a fourth-level resource-based social history course, the rationale was to allow the student to 'be' a researcher by having access to data and resources on the CD-ROM. The course then needed to support students in using these resources. This was reflected in the evaluation which included questions about the student's experience 'as a researcher', the integration of the media and navigation of the resource.
The interactions dimension refers to students' interactions with the software, which are important for understanding more about the learning processes. The outcomes dimension includes students' perceptions and attitudes, which can be crucial in determining whether computer packages are ever used. A wide range of methods can be used to collect data on these different dimensions, including questionnaires, interviews, logs of computer usage and students' work.

Integrative evaluation
This model was developed by Draper, Brown, Henderson and McAteer (1996), members of the TILT (Teaching with Independent Learning Technologies) project for evaluating a diverse range of CAL in use in a university setting. 1 It is a student-centred approach with an emphasis on observations in real classrooms, and aims to improve teaching and learning by enhancing the integration of ILT into the overall situation. It has an outer method and an inner method. The outer method starts with one or more meetings between evaluators, teachers and developers in order to gather information and establish parameters such as the teachers' evaluation goals, the learning aims and objectives to be studied and other features of the teaching situation.
The course material is then examined in order to identify where assessments, learning quizzes or other measures of learning gains could be incorporated and a design for the evaluation is finalized. The model emphasizes a cycle of teacher involvement and the joint establishment of goals, aims and the other central features cited above. The inner method is about the instruments to be used. • The range here is similar to that in the CIAO! framework, including, for example, questionnaires, observations, student confidence logs and interviews. Further information and discussion about these instruments and their use can be found in Draper et al. (1996).

ELT project framework
The ELT project at UNL, funded by BP (Oliver and Conole, 1998) has developed a toolkit to guide staff through an evaluation. It has three main steps: selecting a methodology, gathering data and analysing data. For each of these, the proposed evaluation is reviewed in terms of criteria that help the user to decide which approach to adopt. The final toolkit includes information on applying methods and pointers to literature that discusses their use in evaluations and salient case studies. Again, the focus on resources, assessment criteria, learning outcomes and background information on the courses and subjects is seen as important for inclusion in the evaluation tools, as is the need to involve practitioners.

Principles for evaluation of ILT
These three models have been applied to very different HE contexts. However, they tend to agree on a number of points, which are supported in the literature on evaluating ILT and the wider literature on educational evaluations.
• Many factors are known to affect the successful use of ILT: as these are hard to determine in advance or to control, experimental approaches are not usually appropriate. (The issue of which approach to take is widely discussed in the literature; for example, Gunn (1997), Jones, Scanlon and Blake (1998) and Draper (1997) discuss the problems of trying to apply experimental approaches in this area whilst Oliver (1997) discusses the difficulties of comparative approaches).
• It is important to determine the aims or rationale for the use of ILT in order to decide the appropriate focus for the evaluation.
• Evaluation should involve the learners who are intending to use the technology, and should take account of the whole learning situation. The context surrounding the use of the technology is crucial (Gunn, 1997;Phillips, 1988;. • Practitioners, teachers and designers should be involved throughout the process (see Draper, 1996;Phillips, 1988).
• There should be ways (such as open-ended questions/interviews) of finding out about unanticipated issues (Parlett and Hamilton, 1987).
These guidelines are supported by much of the literature, although, as in many areas, the literature can be contentious. However, the team members were also drawing on their own experience of evaluations over a considerable period of time. These became the principles that informed the development and testing of the evaluation tools developed for use in FE colleges in the Learning with ILT project.

The Learning with ILT project and ILT in FE
The project formed a third part of one of the QUILT programmes mentioned in the introduction and was funded by FEDA. Two phases were already complete and included a literature review, the initial development of a set of evaluation tools, visits to nine colleges and interviews with staff.
The aim of this project, the third phase, was to produce an evaluation tool that would provide FE colleges with a mechanism for evaluating ILT projects. The brief was that this phase should build on phase 2 (which the project team had not been involved in) by applying the tools and methods to six to eight case studies. The timing and resourcing was tightly constrained: the project was to run over a year, to include several dissemination seminars and envisaged to take up to eighty days' work.
The research team that successfully tendered for the project was from the Institute of Educational Technology at the Open University. They had experience of evaluating policy and large-scale implementation of ILT and of research into attitudes towards new technologies (see, for example, Jones, Kirkup and Kirkwood, 1993;Calder, 1994) and of formative and summative evaluations of ILTs in context (for example, Scanlon, Tosunoglu, Jones, Butcher, Ross, Greenberg, Taylor and Murphy, 1998). The team also included two researchers who were not full-time university staff, with experience of teaching, management and research into using ILT in FE.

FE take up of ILT
Following the Higginson report, the QUILT programme aimed to: • support change by individuals and institutions; • reach 50,000 staff in the FE sector and college governors; • provide activities from awareness raising to skills training. Despite such initiatives, by 1998 there was some evidence that the take up of ILT had not been as widespread as hoped and that the use of technology in colleges varied widely between curriculum areas (FEFC, 1998). During the 1980s and 1990s, much of the FE investment in technology had gone towards the building up of central resources (Gray and Warrender, 1995). However, a study by Barnard (1999) of FE biology teachers found that the use of central resources made it difficult for them to use computers flexibly in planning their curriculum work and also led to access difficulties.
Given this apparent gap between the objectives for ILT and the reality in colleges, evaluation has a crucial role to play in finding out what is happening 'on the ground' and informing future policy.

Developing the evaluation materials Challenges and constraints
The main challenge facing the team was how to adapt our experience of evaluation and findings from the literature to this particular context. The 'principles' discussed earlier were all derived from evaluations carried out in real contexts, and were therefore followed as far as possible. However, in this project two particular criteria applied: • Firstly, the tools to be developed were to be used by FE practitioners -managers and teachers in FE colleges working in a pressured environment. Therefore we needed tools that could be used 'off the shelf that did not require a high level of expertise in analysis, and that did not take too much time to use and analyse.
• Secondly, there was a wide range of college initiatives using ILT with very different aims and objectives. The projects involved different subject areas and students of different ages, experience and skill levels. The tools needed to be applicable to all of these situations.
Trying to work within these constraints raised some interesting tensions with the principles. The first criterion was a guiding factor in the decision to base the evaluation tool on questionnaires, and although these can have some limitations (for example, they tend to be 'outcome-orientated') it was felt that they could provide the most flexible methods of gathering data. They could either 'stand alone' or be supported by other sources of information (such as course documents, test results, student logs/diaries) which the respondents might want to cite as evidence for their responses. Questionnaires were also likely to be familiar to managers, lecturers and learners and would not require a high level of expertise to analyse.
The second constraint meant that the project team could not determine the aims of each ILT project in order to decide where to focus the evaluation. Given our second principle -'It is important to determine the aims or rationale for the use of ILT in order to decide the appropriate focus for the evaluation' -this was potentially problematic. A tool was needed that was generic enough to cope with very diverse situations, and yet could take account of the particular context in which it was operating. The decision was therefore made to develop questionnaire templates that could be customized by the users for use in particular evaluations. An additional advantage was that those running the evaluation would need to consider carefully the rationale for the project, and this activity, as well modifying the questionnaire, should allow the evaluators themselves to take ownership of the tool.
The other evaluation principles discussed earlier (in the section on 'Principles for evaluation of ILT') fitted the context of the project well. It had already been decided that six case study colleges would be involved, and this was consistent with evaluation involving the 'end users'. But this principle also emphasizes that evaluation should take account of the whole learning situation, and the context surrounding the use of the technology. Having customizable questionnaire templates allowed for the colleges' own issues to be considered, and open-ended questions allowed for the reporting of unanticipated issues. FE teachers and managers were involved throughout the process, by adopting an iterative cycle of development and modification.
Finally, it was important that the tool should be used! Although questionnaires are reasonably quick and straightforward to use, this was an extra activity in an already full timetable and was most likely to happen if it could also feed into an existing activity or serve another function. We therefore involved a self-inspection consultant who discussed how the tool could help colleges provide evidence about their activities in this area in their self-inspection. Again, this was part of the cycle of development and modification, which is discussed following the next section.

The case study colleges and the projects
Six colleges were involved in piloting the evaluation tools and each college was visited three times. The criteria for selecting the colleges were that they should be enthusiastic about participating and generally working well with ILT, as the tool was at an early stage of development. Within these constraints, the aim was to include colleges in different areas of the country (England and Wales were included), of different size and type (including a sixth-form college) and in urban and rural locations.
The ILT projects in which the colleges were engaged also varied considerably. They included projects on online careers advice; resource based A-level delivery; the educational use of IT for teachers; the use of authoring packages to put materials online; a multimedia key skill project for disaffected 14-16-year-olds and staff development in ILT. The brief sketch below is included to give a flavour of two of the case study colleges: College A is a sixth-form college, with 2,000 students, mainly 16-19 years old. It has three teaching faculties -arts and communication, humanities and business studies, and mathematics, science and technology -and a strong commitment to resource-based learning. Centralized computing facilities include a large learning resource centre and three subject-specific resource bases staffed by library staff and one teacher from the appropriate area.
There are a number of large-scale cross-college ILT projects, two of which were used in trialling the evaluation tools. The member of staff involved in the evaluation project was responsible for the technical aspects of these projects. One project involved careers software and was aimed at 700 first-year students on two-year A-level or advanced GNVQ courses. The students were to work in tutor group sessions, as part of an eight-week series of careers sessions, and the software was designed to replace a one-hour-per-week session by the careers staff. The second project provided Internet access to 100 staff machines and 300 student machines. .The member of staff responsible was keen to evaluate the impact of this policy decision on students' work practices.
College B is one of the largest FE providers in its region, with four sites, 5,000 students and 500 staff. Its wide range of courses includes leisure, construction, art .and design, engineering, business management, and manufacturing technology. It has a number of subject-based workshops and a general IT support unit, available to students on a drop-in basis.
Here the selected project was a multimedia skills project for disaffected 14-16 year olds who attended school but spent one day a week at the college for their GNVQ Foundation Course in Manufacturing. About fourteen students were involved and they used key skills material which had been developed from paper-based versions by an authoring technician using Authorware Version 5.

Involving the practitioners: cycles of development and modification
The approach adopted drew on a multi-phase iterative model of formative evaluation, that had been applied to the development of stand-alone teacher training materials (Jones, Kirkup and Kirkwood, 1993). It also drew on the model that FEDA had previously applied, in particular the use of consultation and dissemination seminars.
Three customizable questionnaire templates were developed: for managers, for lecturers and for students/learners. These would allow the evaluators to collect information from the major stakeholders in the projects and also to triangulate the feedback in order to increase confidence in the findings.
The project had three phases, each of which included visits by the researchers to the colleges, at Which the colleges fed back and discussed their use of and reactions to the questionnaires. In the first phase an initial draft of the tool was developed. Following an initial visit to the colleges to discuss the ILT projects where the tool would be used, feedback was obtained on the tool from the colleges and from FEDA. In the second phase, the tool was modified following feedback on the draft tool and given to colleges to trial. Feedback from the trials was obtained at a second visit to the colleges and a consultation seminar was held to which all the participating colleges were invited. An interim report was produced during this phase and again further modifications were made to the tool. In the third and final phase, the colleges used the modified tools and the data (and the colleges' analyses) was discussed at the final visits. Following analysis of the trials, a draft final report was written and this was sent to participants at the dissemination seminar, who were involved in relevant work in both HE and FE. The dissemination seminar allowed the project to be presented and provided feedback from the participants.

Findings from the development of the evaluation tool
The analysis framework and briefing sheet It became clear from the second visits that the three separate strands of the evaluation process needed more coherence. An analysis template was therefore developed so that the three questionnaires (learner, manager, and lecturer) could be compared and critical areas could be highlighted and cross-referenced.
The second visits also revealed the need for one individual, the manager, to take responsibility for customizing and administering the questionnaires and analysis of the responses. The manager's briefing sheet was developed and outlined the best way to approach the analysis through analysis of the learner questionnaire first and then analysis of the lecture and manager questionnaires together so that learner responses could illuminate the staff responses. The study participants were asked to trial the analysis template before the third visit and their comments were on the whole very positive. Several were surprised at the patterns that emerged and felt that these allowed them to propose important modifications to their ILT projects. The overall message was that the template was robust as it stood -and that problems specific to individual projects should be addressed by customizing.

Other feedback from stage three
The intention was that the colleges would initially use the questionnaires in the second phase of the project, but at this point, only two projects were sufficiently advanced to be evaluated, and the analysis framework and the briefing sheet had not been developed. So whilst the feedback from phase two was useful, it was not very detailed. Full feedback and analysis had to wait until phase three. The rest of this section outlines further findings from this phase of the project.

Using the generic templates and customization
The challenge of using a generic questionnaire to probe into specific situations, persisted, both for the developers and users. For example, the first section in the learner questionnaire included questions about the project (for example, its purpose and how it fitted in). There were problems in formulating questions on 'outcomes' for this section, and these stemmed from the very variable aims of the colleges' ILT projects, which meant that it was difficult to ask specific relatively closed questions about the students' learning experience. Some of the feedback on the learner questionnaire in phase two had suggested that more questions should be included that related specifically to student learning. These were introduced in phase three and included, for example, a question that asked respondents whether 'system X' had helped them to find useful information or practise useful.skills. The intention here, as explained in the guidance notes, was that the term 'system X' would be replaced by the name of the appropriate software or project. However, the feedback revealed that very few of the colleges customized their templates to do this.
Most colleges used the questionnaires as they were and so in these cases this question still referred to 'system X'.
In many instances, the lack of customization was not problematic and the generalized questions addressed the information that the colleges needed. However, in two cases the participants realized when they reached the analysis stage that the key questions had not been asked. This highlights the importance of customizing -in particular with the learner questionnaire -so that colleges can address the specific aims and objectives of their projects. As a result, the final manual outlined the importance of customization and, in particular, of ensuring that the questionnaire reflected the specific aims of the project being evaluated.
Another finding related to this issue, was that colleges had different experiences in using the questionnaire: sections and approaches that were generally found helpful were occasionally seen as having limited importance by particular colleges (for example, access) as they were not sufficiently salient to the project or the college. Again, customization would address this.

Different stakeholders' experiences
The manager questionnaires were completed thoroughly and they reported that they had found the process of thinking through the strategic aims salutary -often projects had been introduced so quickly that teams had not had time to reflect on exactly what they were trying to achieve. By contrast, lecturer responses varied considerably in their quality and reflected some resistance to using ILT. It seemed that those questionnaires that were completed more thoroughly reflected good communication and/or promoted good communication between managers and learners, and a shared ownership of projects. One manager in particular noted that the evaluation process had been invaluable in acting as a framework to support discussion and develop coherence within his team. Students sometimes had difficulties with answering questions about the purpose of the ILT activities. The lecturers had also had similar difficulties in phase two of the project where they had left the strategic aims section blank in their questionnaires. In the final version, in section three, rather than asking what the strategic aims were, it was decided to ask whether they had seen the strategic aims. This also yielded information about their involvement in and ownership of the project.

Discussion and implications
As indicated above, the anticipated customization did not happen. Time may have been a factor: it was a major constraint and throughout the project the team was aware that the colleges were finding it difficult to keep up with their planned implementation timetables. Also, the lack of customization may reflect the fact that the same individuals were involved in successive trails and on the first round we had asked them not to customize the templates.
Both lecturers and learners were at times unsure about the purpose of the projects they were evaluating, and, as discussed earlier, we could not include questions about each project's particular objectives because each was different. Three factors may be relevant to the lecturers' difficulties in answering questions about the aims of the project. First, there was some feedback that the design of the questionnaire needed modification and it was modified during the project. Secondly, such problems sometimes reflected the lecturers' lack of ownership and understanding of the projects particularly where respondents had joined the project once it was under way and had not been involved with setting it up.
Thirdly, a few lecturers appeared to have more general difficulties in articulating aims and objectives.
The process of trialling revealed the need for the person overseeing the evaluation process to be responsible for customizing the questionnaires and to have a strong awareness of the aims and objectives of the project.
The feedback from the colleges strongly supports the principle that users must be involved in the development cycle. Without such an involvement, crucial information such as the difficulties that colleges had with determining the aims and objectives of the projects and the importance of customization (and that colleges often did not readily do this) would have been missed. There were also a number of significant changes between the first and final drafts of the learner questionnaire templates, which supports the importance of iterative trialling in this way, and the final version of the manual stressed that customizing is crucial to the success of the evaluation procedure.

Conclusions
When new approaches to teaching are adopted, evaluation is particularly important and although it has often been an 'afterthought' in the past, evaluation is increasingly viewed as part of any new project. Other changes are also under way. Whereas an investigation of an innovative project may previously have been carried out by outside evaluators, with the current emphasis on quality assessment, teachers in post-compulsory education are likely to be increasingly engaged in evaluation. It is likely that such teacher-evaluators, unlike external evaluators, will not have any appropriate research or evaluation experience and are certainly likely to be short of time.
The Learning with ILT project's aim was to develop a generic evaluation tool that could be used in these circumstances. Whilst the constraints were challenging, it did prove possible to follow the guidelines outlined earlier and to produce a tool that was usable. However, we would emphasize the need for tailoring if a generic tool is to be used successfully in very different contexts.
It is also important that the need for support when starting to use evaluation tools such as this should not be underestimated. The case study colleges were keen to take part, and had the 'spur' of visits from the researchers to keep them going. Even so, time was a major constraint, so it would be very easy for colleges that did not have the process timetabled in this way to let other priorities take over. Guidance in using the evaluation tool as part of the self-inspection process was important in providing a rationale and motivation for using the tool. It is too early to know whether colleges will take on board such evaluation processes once they are on their own, but it is clear that having another reason for doing the evaluation is particularly important when resources are limited.