Driving departmental change through evaluation : some outcomes and problems

This paper identifies the ways in which a three-year technology-based learning and teaching project has addressed the issue of catalysing departmental change. In order to promote change at this level, it is necessary to relate the accepted learning and teaching parameters of specific disciplines to meaningful evaluation data of student and staff perceptions, in an attempt to broaden the understanding of academic staff. Thus, a number of factors become important to the process of change including: supportive role models within departments; forging feasible departmental implementation plans; utilizing support staff with a technical and pedagogical awareness; and fully involving students in curriculum development and design. Inhibitors to change also need to be noted, for instance: time-management; resource and training allocation; and a lack of managerial support. However, the creation of a supportive structure which highlights good practice is fundamental in gaining uptake of materials and in changing perceptions. An acceptance of staff development needs in the light of the objectives of both the student and the curriculum is required This holistic approach provides a suitable environment for the symbiosis between learning and teaching to develop.


Introduction
This paper will focus upon some of the outcomes from the first year of the Courseware for History Implementation Consortium (Chic), a Teaching and Learning Technology Programme (TLTP), Phase 3 project.It is particularly important that the consortium addresses the issue of embedding change within organizations if it is to succeed in its aims.However, as Brown (1998) has noted, the integration of learning and teaching innovation into the broader fabric of an institution, by projects with a fixed remit and life cycle, is problematic.Here we will investigate how the Chic project has related evaluation outcomes to disciplinary pedagogical imperatives in order to foster a greater sense of purpose and cohesion towards the project and its objectives.This recognizes the value of evaluation in affecting the context of learning, through the iteration of teaching techniques and resource provision.The resultant implications for institutions, and the barriers to change, need to be examined in the light of the pedagogical context of the history discipline.Whilst accepting the premise that change needs to be placed in the context of institutional policy (Hart, Ryan and Bagdon, 1999;Li and Kaye, 1999), this paper will show how a culture of change can be promoted within departments.

Pedagogical context
The project's primary aim is to develop and disseminate distinctive pedagogies, through the production of context-specific case studies, to support the use of existing TLTP courseware and other learning-technology-based materials within the mainstream history curriculum of UK higher education.These pedagogies should enhance the quality of student learning beyond what might be achieved solely by the use of traditional teaching methods.
To date, apart from the History 2000 project (Fund for the Development of Teaching and Learning, project 20/96), little cohesive research has been undertaken into the learning and teaching of history at undergraduate level.History 2000 has taken an enlightened approach to the development of learning and teaching within the discipline, noting that 'C & IT-related work should be one important thread in a multi-stranded and co-ordinated strategy for developing learning and teaching as a whole'.Moreover, it states that there is a need for 'examples of how C & IT can be embedded in current structures and modes of teaching and learning' (History 2000 Response to the HEFCE Circular Letter 36/98,1998).
In order to assess and address the factors which affect pedagogy, and to promote the active engagement of students in the practice of history, the Chic project staff need to understand how effective student learning can be hindered and promoted.Measuring the reactions and perceptions of students and staff, and mapping these onto departmental imperatives is crucial.To address the concerns of teaching staff and managers, it is important to examine whether the integrated Web-based materials are sufficiently transparent that they promote deep-level learning which then fosters critical reasoning and autonomous self-reflection within the discipline (Rogers, 1969;Sharp, 1990;Bowden and Marton, 1998).If so, then departmental change becomes justifiable.Explicitly making the link between pedagogic innovation and educational development, and meaningful evaluation data is vital in raising awareness and underpinning departmental change.

Evaluation context
The evaluation of the Chic project has both to address generic learning and teaching issues, and to acknowledge the diversity which exists within its seven consortium sites, or subprojects.The consortium consists of three pre-and three post-1992 universities and one university college, all of which have vastly differentiated levels of technical support available.In the 1998-9 session, the evaluations focused on modules with fully-and partially-integrated and customized ICT elements.These modules involved between 2 and 250 level one, two and three students, both part-and full-time, and of all ages.
Despite this heterogeneity, one of the main objectives for the first year of the project was to establish a comparable and viable framework for evaluation that would produce meaningful research data.The creation of context-sensitive case studies requires the use of thoughtful qualitative methods, which allow one effectively to evaluate and assess the value of ICT in the learning and teaching of history, whilst recognizing diversity.A common core of questions and issues has been addressed throughout the consortium, and generic pro forma information was generated which allows comparisons to be made across types of student or course.
However, each site has to match its evaluation tools to its specific implementations.The key here is the relevance of the questions or issues raised to student and staff perceptions of their local infrastructure, customization and delivery mechanisms, and how these are perceived to have affected learning experiences and outcomes.Learning outcomes and processes will be further analysed through an investigation of assessment scores, by observing student learning patterns in laboratories, by examining student diaries, and through sensitive interviewing.The triangulation of these mechanisms will allow the project to develop an increasing understanding of possible learning gains in different contexts over its three-year duration.
Clearly, staff resourcing issues affect the levels and types of evaluation data which can be collected.In general, the sub-project co-ordinators managed the local evaluative process, although module tutors, fully-briefed graduate teaching assistants and the project manager also conducted interviews and observations, ran focus groups and delivered questionnaires.It was considered crucial that the module tutor was seen, by the students and staff, to be committed to the purpose and process of evaluation, although in some instances it was felt that s/he was too close to the material effectively to evaluate it, and so an external evaluator was utilized.Table 1 relates the different sites, the module levels where ICT implementations were taking place, and the total number of students on those modules, to the methods used to evaluate student perceptions, and the relevant numbers of students involved in those methods, in the first year of the project.(Under site type, UC represents university college, Pre-92 represents a university created before 1992, and Post-92 one created after 1992.)Table 2 shows the numbers of staff involved in implementing innovations in modules at each site, alongside those undertaking the evaluation, and the evaluative methods which were utilized.

External examiner's views
The process of evaluation was complex, affecting people with different priorities.It has also been iterative and provided a more coherent strategy for the rest of the project's life.Moreover, it has allowed each site to generate findings which will feed into future learning and teaching plans.Increasing awareness about these evaluations has been fundamental in allowing the project to feed directly into departmental strategies.

Key evaluation findings and their implications for departmental change
In order to create a climate where departmental change can flourish it is crucial that the symbiosis between learning and teaching is contextualized.This requires the triangulation of staff and student findings in order to illuminate the possibilities for pedagogic innovation, and for the promotion of departmental change.

The student experience
The student evaluations highlighted several issues that impinged upon learning.These were the need for: • integrated materials; • locally customized resources that provide differentiated learning pathways; • improved access to materials; • relevant tutor-led training.
Generally, the students valued an integrated pedagogical approach using a range of traditional and online materials.Second-and third-level students commented upon the greater understanding they gained through primary evidence which 'brought you closer to history'.This understanding was aided by the fact that locally customized TLTP courseware allowed them 'to develop [their] own ideas', and that the range of sources and their ease-of-comparison 'made one's approach to the sources more flexible'.Clearly, customization that matches resources to learning and teaching processes is important in generating student interest.However, it is clear that staff need to come to terms more fully with their students' learning styles and contexts.A majority of level-one students, alongside many who were mature and/or part-time, did not see computer-based materials making a positive difference to their learning.The prior experiences of these learners and the lack of customization for specific modules at certain sites gave these materials the look of any other pre-packaged resource to be used or ignored.One first-year noted that the courseware 'contained so much material that it is very easy to become swamped, and therefore miss the main points of the documents'.Students with limited research and critical skills want and need more direction, which focuses the need to align customization to the prior experiences of the students, and their levels of critical ability.This in turn impinges upon pedagogic imperatives.
Such imperatives are also affected by access to relevant materials.As has been argued by Trathen and Sajeev (1996), there needs to be access from as many sites as possible if any student's educational well-being is to be supported.One second-level student found 'the computer room often too noisy to concentrate', whilst one of her peers noted that 'I disliked the environment for accessing the tutorial, i.e. a busy computer room'.This is particularly relevant because, for a majority of the students who were interviewed across the consortium, time-management was very important.Two full-time level one students were not attracted to using the materials in college, where they found the machines substantially slower than their own computers.
At every site some students had difficulties in gaining access to the computer resources or in satisfying their own learning pathways.Those living off-campus or learning at a distance, and those who did not work well in the context of a computer room, needed access from home or from machines in open-plan rooms, with space to begin to construct their own perceptions of a historical period or argument.Clearly, there is a need to balance the ICT and resource priorities of departments and institutions, bearing in mind that technical difficulties are the quickest way to disaffect both students and staff.
Of equal importance for students was tutor support in mastering the technologies that would be used to achieve the course objectives.Basic IT courses, either run by the history department concerned or by institutional IT support or education units, were available for all students involved in the project.However, purely voluntary sessions without formal requirements were not taken up by many, and this has important ramifications for firstyear and mature students.These learners needed enough technological guidance from the tutor who was leading the course, in supervised, timetabled sessions, to remove any barriers to learning that the computer interface may have presented.Having the historical and technological skills was crucial.It was only where the technology became sufficiently transparent that positive learning outcomes, in terms of attitudinal changes and deep-level thinking, were perceived by students and staff.If departments are successfully to promote learner-autonomy, they need to understand and act on these issues.

Staff perceptions
The themes generated by the student findings demand adequate context-based, holistic educational development for staff, in order to give them technical and pedagogical support (Biggs, 1999).How far this can be achieved without reorganizing the teaching programme is unclear, and in any case this may not affect the negative perceptions which are held by many staff about the role of ICT.These negative reactions are broad-based and interconnected, having personal, academic and technological focuses.
Personally, skills are a problem for some staff who felt that they were not as competent as their students.Support for academic staff is crucial but institutions must overcome problems concerning the time to invest in training and teaching reorganization.One tutor noted that 'if training was an add-on it would be seen as an unnecessary extra burden'.However, addressing the basic need for staff to understand how computer-based resources can add depth to student learning requires that they see their courses in terms of their student's needs.Approaching course development in this way is essential in order to overcome student reluctance.
Academically, staff fears tended to focus on: plagiarism; the undermining of a reading culture; the lack of a relevant critical approach; the pre-digested, inflexible or formulaic approach of the courseware; and student concerns about the loss of tutor and peer-group support and guidance.These issues highlight the need for integrated technical and pedagogical staff support which will allow ICTs to be integrated more fully into teaching and learning, and make staff aware of the possibilities for creating differentiated yet relevant materials within a supportive learning context.One academic believed that online materials could alter student's learning patterns depending upon their enthusiasm, adding that 'attractively packaged materials undoubtedly sparks/develops interest and hence stimulates learning'.
Technical factors were major causes of staff reticence, even amongst the IT-literate.Whilst one member of staff at a pre-1992 university stated that the tutorials were 'a good process of integrated learning', he added that he would not push such learning 'until I know that the machines are there and that the students have access'.The six staff who taught at satellite sites or who worked off-campus also pointed to infrastructural difficulties, such as networks crashing, which made effective learning, teaching and research difficult.When coupled to the limited hours available to students in some computer labs, the lack of a robust system is a major problem.The concomitant pressure from students both for greater availability of materials and better facilities creates a demand for departmental action.
These factors have important ramifications for those staff who are reticent or opposed to the use of ICT in their teaching.Such staff need support because departments cannot make training voluntary while expecting the successful integration of ICT-based materials.This is particularly pertinent as several students warned that they 'would not have chosen the course if it revolved solely around it [the materials]'.However, for some staff the nature of the language of educational development is beyond their discipline-situated compass and at times anathema, creating a barrier to involvement.This tends to add to a sense that many staff are not being touched by institutional learning and teaching development units.Of importance here is finding methods by which departments can invest in in-house developments, and make the link between evaluation and pedagogy explicit.

Departmental change mechanisms
Driving departmental change These student and staff evaluations illuminate disciplinary preconceptions with respect to pedagogical innovation and underpin the mechanics of departmental change.There is a need to match the staff's personal, academic and technical concerns to the integration, customization, access and training issues raised by the students.Across the consortium, there is a wide range of staff involvement, both in terms of teaching, training and full awareness-raising, and several strategies have been employed to promote departmental innovation in the use of ICT.There has been no one way of achieving this, but a mix of methods has been utilized.
Both management and staff noted how 'the power of role models' is fundamental in forging an awareness of good practice, and in allowing departments to overcome the tensions created by attempting the integration and customization of materials.Each of these sites had IT-literate academics as their sub-project co-ordinator.Such staff are crucial in overcoming the fears of those who felt that they were not as IT-competent as their students.Utilizing both individual and group demonstrations these academics have, according to one dean, made 'colleagues . . .much more aware of [the] potential represented by ICT for effective teaching in history'.At one site it was vital that the coordinator found time to work in one-to-one situations with those who were nervous about their skills levels or ambivalent about the worth of computer-based resources. .At two other sites, dedicated ICT away days have been used to share the findings of the consortium's evaluations.For one dean circulating such responses was crucial as 'positive student responses have confirmed the merits' of ICT-integrated initiatives.Two heads of department at different sites concurred.The first noted that such initiatives could raise departmental awareness, whilst the other added that 'it has helped show doubting Thomases that this ... is feasible and within the potential of both staff and students'.These approaches work to strengthen or even alter a department's approach.
Two consortium history departments already had well-developed and embedded ICT strategies.One of these, along with a third, also had dedicated ICT support staff attached to their faculty.These staff had time to forge a positive link between the technical and pedagogical needs of the academics, who had some of the pressures on them removed, by: finding time to deal with ICT services to get work undertaken; creating effective and relevant individual pathways for staff re-skilling; and, understanding how ICTs can be successfully integrated within the fabric of a course.The construction and negotiation of an active and empowering dialogue between ICT staff with a pedagogical understanding and academic staff with a range of technological expertise allowed these departments, and particular staff within them, to consider thoughtful curriculum change and adaptation with confidence (also exemplified by Stephens and Dalgleish, 1999).
The process of wider staff involvement and the sharing of research data has directly informed the planning of four consortium departments.Using a mixture of group and individual demonstrations, general awareness-raising and dialogue about learning and teaching, these departments have been developing implementation plans for next semester, in history modules, joint history-English modules and in a cognate discipline -art and design.This is a very positive outcome and underlines the importance of supportive environments in effecting change.These implementation plans have become levers to generate wider interest within and across departments.
For instance, six of the staff at one site will be using materials with between 25 and 100 students at levels one, two and three.Not only will the training sessions be compulsory, but there has also been some sense of rethinking the curriculum and course design.Source analysis, workshops and commentaries will form significant parts of the module assessments.It is important to note the statement made in this department's implementation plan that 'with a radical review of the history curriculum forthcoming, made necessary by impending retirements, the possibility of explicitly designing course(s) around TLTP courseware was discussed.It was agreed to return to this matter once further evaluation had been carried out.'The methods that succeeded in driving departmental change this far will be utilized to maintain the momentum for improving the processes and contexts of their students'learning.
In a separate department an on-going focus group held at regular intervals with first-year students empowered both students and staff to work together in a manner akin to a learning development unit.The students made suggestions about course and Web-site design, and courseware customization, as well as advising on training and evaluation strategies.A module Web site was evaluated by them and the involvement of the Arts Faculty ICT Officer was vital in giving the students the confidence to express their own technical needs.This sense of collaboration meant that their suggestions, mediated through academic and technical staff, were taken very seriously in designing teaching strategies for the 1999-2000 academic year.
A process of redesigning teaching strategies is being undertaken at four sites, and is being focused upon module Web sites linked to both the courseware and discussion lists, along with further customization of materials.Two of these departments are also purchasing their own servers for in-house materials in an attempt to improve access for students.Integrating resources was important for the deans of faculty at these sites who saw benefits accruing from having materials and handbooks on an intranet, with open access for all students.One dean noted: "The matching of the electronic materials to delivery systems on site here has not been trouble-free.However, development of [the] intranet will alleviate this problem'.Such outcomes indicate a clear sense of pedagogic drive fostered by meaningful evaluation, which is fundamental to the process of change (Draper, 1997;Mcllvea, Greenan and Humphreys, 1997).
There were also strategic drivers for change.One dean felt confident that such developments 'will be regarded favourably in future QAAHE reviews'.Two senior managers stated that quality assurance is clearly an important factor, as it recognizes the need for wider use of ICT in higher education, although one added that in the implementation of computer-based learning there is a need to recognise the academic culture of departments and the autonomy of staff.These perceptions were matched by those of four heads of department or course leaders, who felt that the use of ICT would help to attract students as well as contributing to favourable subject review.One noted: 'We want to be seen as a cutting-edge department in all aspects of teaching quality and innovation.ICT helps.'Another was more tentative, and stated that while skills and employability were parts of their recruitment strategy, their possible effect on numbers was 'more a hope than anything else'.

Inhibiting departmental change
Despite the fact that the Chic project has found an almost overwhelming student acceptance of the principle that teaching materials should be available online, there are broader institutional issues that affect departmental change, and these mirror those highlighted by the Atkins report (HEFCE, 1998).The stresses on an individual's timemanagement and the ability to find money to devote to infrastructure are crucial.One head of department was worried that 'extra work for some in the developmental stage was not easy to manage'.A separate department was having to contend with a business ethic which ran through its institution's ICT support services.This confronted the department's wishes to develop its ICT provision within a small budgetary constraint, with a need to purchase support that was calculated by individual network connections.For large departments with low ICT budgets such policies could prove crippling.
Clearly, the provision of adequate time and infrastructure is vital in creating a proactive culture of change.As important is the-understanding of managers and their ability to act to make staff aware of the potential of online materials.This demands that the whole issue of ICT needs to be placed firmly within the compass of institutional learning and teaching policy (Carrotte, 1994;Foley, 1994).In proactive departments the lack of an institutional lead can be overcome.However, where there is no strong top-down pull towards pedagogical innovation, in essentially reactive environments, with technophobic or hostile senior and middle managers, ICT-based projects are likely to fail.Overcoming a lack of overt support, which is often based on a lack of awareness about what is required beyond the short-term, requires communication and dialogue within faculties and departments about ICT in the context of learning and teaching, illuminated by evaluation data.

Conclusion
From the perceptions of the students and staff involved in its first year, the Chic project has shown that the use of integrated computer-based materials can encourage the development of critical and independent thought.A major issue for the second year of the project is to address the extent to which students want to be empowered to follow their own learning techniques rather than being directed too closely, and this impacts on issues of service-provision (Hill, 1995;Rowley, 1997).It is vital that differentiated learning is supported and in this way the development of departmental implementation plans, with the support of middle and senior managers is fundamental.
Such change has been driven by a series of factors which in some combination have supported innovation.These factors were: • role models at each site who could work on a one-to-one, and one-to-many basis; • holistic, context-specific staff training and dedicated staff away days; • illustrating and supporting innovation with student evaluation data; • matching evaluation data to broader pedagogic perceptions; • forging departmental implementation plans that support curriculum change and successful QAA review; • effecting positive links between technical and teaching staff.
The success of these factors in supporting innovation is affected by resource implications and management priorities.However, departmental change requires a holistic approach from the change-agent and this is why the Chic project is attempting to address the needs of all stakeholders.Without support from students,, staff and managers change will be illusory.
The finance and support that the consortium sites enjoy has made these developments possible.Beyond the life cycle of the project, the developments are likely to continue because their worth to the departments and faculties involved is explicitly acknowledged.How this is then transmitted to the history teaching fraternity may depend upon the role of the subject centre, the practice of collaborative networks of academics, and institutional policy.However, the Chic project has shown how the creation of a supportive structure which highlights good practice is fundamental in gaining the uptake of materials and in changing perceptions.Such an approach can provide a suitable environment for the symbiosis between learning and teaching to develop.